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ABSTRACT 

i 

The  purpose  of  the  study  described  in  this  report  was  to  investigate  the  | 

design  engineer’s  concept  of  the  relationship  betveen  system  characteristics  J 

ari  various  skill  dimensions.  Fourteen  paper  and  pencil  tests  specifically  I 

developed  to  examine  these  relationships  were  administered  to  eight  design  1 

engineex-s  during  t'.'C  four-hour  sessions.  Design  characteristics  significantly  i 


related  to  sKill  level  are  test  points,  internal  components,  checkout  and  § 
troubleshooting  procedures,  type  of  test  equipment  required  and  go/no-go  dis-  | 
plays.  Individual  design  concepts  such  as  component  repair,  etc.,  are  also  I 
significantly  related  to  the  amount  of  training  required.  The  ergixxeer's  con-  f 
cept  of  skill  level  is  more  performance -oriented  than  that  described  by  the  | 
Air  Force  Specialty  Code  designations.  It  vas  found  that  the  engineer  con-  f 
ceptualises  maintenance  skill  in  terms  of  knowledge,  troubleshooting  ability  | 
and  flexibility.  A  common  denominator  of  skill  parameters  appears  to  be  j 


troubleshooting  ability.  Skill  level  appears  net  to  be  related  in  the  engineer's 
mind  to  years  of  experience. 


SUMMARY  AND  CONCLUSIONS 


PROBLEM 

Previous  studies  indicate  that  design  engineers  have  great  difficulty 
in  understnading  and  utilizing  Air  Force  descriptions  of  personnel  skill. 
As  a  result,  their  designs  often  do  not  reflect  adequate  consideration  of 
skill  requirements.  The  present  study  sought  to  determine  the  understand¬ 
ing  that  design  engineers  have  of  personnel  skill,  and  how  they  relate  it 
to  system  design  characteristics. 


APPROACH 

Fourteen  paper  and  pencil  tests  specifically  developed  to  examine  these 
relationships  were  administered  to  eight  design  engineers  during  two  four- 
hour  sessions.  A  set  of  skill  parameters  and  design  characteristics  was 
developed  which  hypothetically  different iateu  highly  skilled  maintenance 
personnel  from  those  less  skilled.  Subjects  were  asked  to  categorize 
sample  critical  incidents  illustrating  these  parameters  and  characteristics 
in  two  ways:  as  (1)  engineers  conceptualize  skill  differences;  and 
(2)  as  engineers  consider  that  skill  would  influence  their  hardware  design. 


RESULTS 

The  engineer  relates  a  number  of  design  concepts  and  characteristics 
such  as  test  points,  internal  components,  checkout  and  troubleshooting 
procedures,  type  of  test  equipment  required  to  the  skill  level  of  the 
maintenance  technician.  Individual  design  concepts  are  also  significantly 
related  to  the  amount  of  training  required.  The  engine _r  conceptualizes  • 

maintenance  skill  in  terms  of  knowledge  of  the  system,  troubleshooting  • 

ability  and  flexibility,  with  tne  amount  of  troubleshooting  ability  emerg-  j 

ing  as  the  common  denominator  of  the  various  skill  levels  studied.  The 
engineer's  concept  of  skill  level  is  more  performance  oriented  than  that  - 

described  by  Air  Force  Specialty  Code  designations.  Skill  level  appears  «; 

not  to  be  related  ir.  the  engineer's  mind  to  years  of  experience. 


CONCLUSIONS 

Design  concepts  and  characteristics  are  related  in  the  engineer's  mind 
to  different  skill  levels.  Given  a  particular  skill  requirement,  the 
engineer  will  include  in  his  design  those  characteristics  that  he  considers 
appropriate  to  that  skill  requirement.  The  engineer  has  a  concept  of  skill 
level  which  is  more  operationally  and  performance  oriented  than  described  by 
the  Air  Force  Specialty  Code  designations,  with  troubleshooting  ability  as 


;l 


the  generalized  factor  underlying  all  level?  of  skill.  This  suggests  that, 
for  the  designer,  skill  level  might  meaningfully  be  defined  as  "degree  of 
troubleshooting  ability."  Individual  design  concepts  are  significantly 
related  in  the  engineer's  mind  to  the  amount  of  training  required.  A  speci¬ 
fication  of  the  amount  of  training  to  be  given  to  system  personnel  will 
therefore  influence  the  type  of  design  concept  incorporated  into  new 
equipment. 


OPERATIONAL  IMPLICATIONS 

Manpower  requirements  included  in  procurement  documents  should  be 
phrased  as  explicit  design  requirements.  The  maximum  number  of  personnel 
who  will  operate  and  maintain  the  system  and  the  highest  skill  level  of 
these  personnel  should  be  described.  Skill  level  should  be  phrased  in 
performance  terms,  and  the  design  implications  of  manpower  requirements 
should  be  explicitly  stated  as  desired  outputs  of  system  design. 


RESEARCH  IMPLICATIONS 


It  is  recommended  that  a  "human  resources  data  system  development 
handbook"  be  developed  which  would  describe  the  various  classes  of  equip¬ 
ment  and  their  design  characteristics,  together  with  the  manning/skill 
dimensions  required  to  operate  and  maintain  that  equipment,  applicable 
training  data,  and  characteristic  task  behaviors.  Such  a  handbook  could 
be  constructed  by  (1)  developing  a  taxonomy  of  design  concepts  and  character¬ 
istics  which  differentiate  various  types  of  systems,  (2)  selecting  opera¬ 
tional  systems  in  the  Air  Force  which  are  representative  of  these  classes, 

(3)  going  to  the  air  bases  at  which  these  systems  are  to  be  found,  and 

(4)  performing  an  analysis  of  the  task  behaviors,  skill  parameters  and 
design-skill  relationships  demanded  by  these  systems.  Thi*  handbook  could 
then  be  provided  to  Air  Force  managers  and  design  engineers  so  that,  given 
a  requirement  to  develop  a  system  of  a  given  type,  they  could  retrieve  the 
appropriate  equipment  design  type  and  find  the  manning  dimensions  and 
design  relationships  required  for  the  new  design. 


vi 


•*- 


CONTENTS 


II. 

III. 

IV. 

REFERENCES 


INTRODUCTION 

A.  Nature  of  the  Problem 

B.  Purpose  of  the  Study 

TEST  METHODOLOGY 

A.  General  Strategy 

B.  Subjects 

C.  Test  Description 

RESULTS  AND  CONCLUSIONS 

A.  Summary 

B.  Detailed  Results 
c*  Conclusions 

RECOMMENDATIONS 


1 

1 

2 

C 

s 

5 

6 
6 

11 

11 

13 

61 

62 

65 


vii 


LIST  OF  TABLES 


Table 

Title 

?a§e 

1 

Subject  Education  and  Experience 

7 

II 

List  of  Tests  Administered  to  Subjects 

III(A) 

Results  of  Test  C  (Skill  vs.  Design  Characteristics) 
With  Original  Categories 

16 

Ili(B) 

Results  of  Test  C  (Skill  vs.  Design  Characteristics) 
With  Collapsed  Categories 

20 

IV 

Results  of  Test  E-2  (Skill  vs.  Automation  Test) 

24 

V(A) 

Results  of  Test  B-3  (Skill  vs.  Design  Concepts) 

For  Unskilled  Personnel 

25 

V(B) 

Results  of  Test  E-3  (Skill  vs.  Design  Concepts) 

For  Skilled  Personnel 

26 

VI 

Results  of  Test  G  (Design  Suitability  Test) 

29 

VII 

Results  of  Test  F  (Design  Independence  Test) 

3^ 

VIII 

Results  of  Test  D  (Design  Tradeoff  Priorities  Test) 

37 

IX 

Results  of  Test  I  (Design  vs.  Training  Test) 

40 

X 

Results  of  Test  A  (Skill  Dimension  Test) 

U2 

XI 

Results  of  Test  3  (Skill  Importance  Test) 

51 

X1I(A) 

Reeults  of  Tsst  H  (Skill  Ranking  Test) 

5** 

m(B) 

Results  of  Test  B  (Skill  Importance)  Compared  with 

Test  H  (Skill  Ranking) 

55 

XIII 

Resilts  of  Test  J  (Skill  level  Differences  Test) 
Distribution  of  Judgments 

58 

XIV 

Results  of  Test  E-l  (Skill  vs.  Experience  level  Test) 

6o 

viii 


SECTION  I 


INTRODUCTION 


A.  NATURE  OF  THE  PROBLOi 

Manpower  in  system  development  and  operation  can  be  described  in  terms 
of  two  basic  dimensions:  number  and  skill  level.  The  human  resources 
specialist  seeks  by  means  of  various  analyses  and  inputs  to  the  system 
development  process  to  optimize  the  number  and  skill  level  of  personnel 
required  to  operate  and  maintain  the  system  against  such  factors  as  avail¬ 
able  human  resources,  training  costs,  personnel  maintenance  costs,  human 
contributions  to  system  operations,  etc.  (Eckstrand,  Askren,  and  Snyder, 

1967;  and  Snyder  and  Askren,  1968). 

To  secure  this  desirable  condition  means  that  systems  must  be  designed 
so  that  they  can  be  operated  and  maintained  by  the  desired  personnel  force. 
Although  it  would  be  desirable  to  accomplish  this  end  simply  by  requiring 
that  systems  be  designed  in  this  way,  it  is  impossible  to  do  so  without 
first  describing  to  the  engineer  the  relationship  between  personnel  number 
and  skill  level  and  the  design  characteristics  which  the  engineer  incor¬ 
porates  in  the  new  system. 

That  relationship  operates  in  two  ways:  the  imposition  of  a  manpower 
requirement  may  force  the  design  engineer  to  consider  alternative  design 
concepts;  in  the  other  direction,  the  manner  in  which  he  designs  his  system 
may  require  that  certain  numbers  and  types  of  personnel  be  supplied  to 
operate  and  maintain  that  system.  Thus  one  can  look  at  the  personnel  design 
relationship  in  terms  of  the  effect  of  personnel  on  design  and  the  effects 
of  design  on  personnel. 

In  the  first  case  it  is  necessary  to  know  in  advance  what  the  imposi¬ 
tion  of  a  manpower  requirement  will  mean  to  design,  in  order  to  avoid 
constraining  the  engineer  unreasonably.  In  the  second  case,  in  order  to 
predict  the  number  and  type  of  manpower  to  support  the  new  system,  it  is 
necessary  to  determine  the  implications  of  various  types  of  design  for 
the  personnel  who  must  be  provided. 

At  the  present  time  there  exist  no  firm  data  concerning  these  peisonnei- 
design  relationships.  Because  it  is  difficult  to  predict  the  effects  of  a 
manpower  requirement  on  design,  procurement  agencies  have  been  reluctant 
to  specify  concrete,  stringent  manpower  requirements  in  system  procurement 
documents.  In  consequence,  since  engineers  tend  to  ignore  generally  worded 
requirements  which  lack  concrete  design  implications,  manpower  requirements 
have  In  the  past  usually  failed  to  have  any  meaningful  effect  on  design. 

A  previous  study  by  Me is ter,  Sullivan  and  Askren,  1968,  indicated  that 
engineers  have  great  difficulty  in  translating  Air  Force  skill  designations  (e.g. 
3,  7-levels)  into  meaningful  design  terms.  The  concept  of  number  of 
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personnel  is  relativ. ly  "lambiguous ,  because  it  is  uni -dimensional  and 
quantitative.  On  the  other  hand,  skill  can  be  conceptualized  in  many 
dimensions,  such  as  intelligence,  aptitude,  sensory  capability.  Three 
questions  thereiore  arise: 

(1)  What  dimensions  are  actually  represented  in  the  engineer’s  concept 
of  skill?  In  order  to  impose  skill  requirements  on  engineers,  it 
is  necessary  to  describe  these  in  terns  of  dimensions  vhich  are 
meaningful  to  the  engineer. 

(2)  How  are  these  dimensions  described  in  performance  terms* 

(3)  What  are  the  implications  of  these  skill  dimensions  for  design  (ie, 
what  design  characteristics  are  most  appropriate  for  different  levels 
of  skilled  personnel)? 

The  previous  study  by  Me is ter  et  al.,  1968  recommended  further 
investigation  into  the  skill  problem.  The  study  described  in  this  report 
was  an  initial  effort  to  specify  the  relationship  between  personnel 
characteristics  (basically  skill)  and  equipment  design  characteristics , 
as  this  relationship  is  viewed  by  the  engineers  who  rust  translate  per¬ 
sonnel  characteristics  into  design  concepts. 


B.  PURPOSE  OF  THE  STUDY 

The  overall  purpose  of  the  study  was  to  investigate  the  relationship 
between  skill  dimensions  and  equipment  design  chcracheri sties.  Such  a 
relationship,  however,  has  many  facets,  each  of  which  must  be  examined 
closely.  For  example,  the  nature  of  the  engineer’s  reaction  to  a 
personnel  skill  requirement  depends  on  how  he  conceptualizes  that  re¬ 
quirement;  in  other  words,  how  he  defines  to  himself  what  skill  means. 

It  is  therefore  necessary  to  determine  what  skill  dimensions  are  considered 
by  engineers  to  differentiate  different  levels  of  maintenance  technician 
skills. 

In  order  to  explore  how  skill  requirements  affect  design,  it  is 
necessary  first  to  determine  the  different  ways  in  which  the  engineer 
tends  to  design*  This  means  that  it  is  necessary  to  determine  the  range 
of  design  concepts  which  can  be  affected  by  the  skill  requirement. 

The  relationship  between  design  and  skill  can  be  understood  only  in 
terms  of  hew  engineers  view  specific  design  characteristics  and  concepts 
to  be  related  to  the  various  skill  dimensions  and  skill  levels.  Design 
too  has  many  facets.  What  aspects  of  design  are  related  to  vhat  aspects 
of  skill?  This  too  must  be  investigated. 

Just  as  skill  may  have  e  number  of  different  dimensions,  so  too  it 
probably  has  a  number  of  dif.  rent  levels.  Obviously  the  nature  of  the 


2 


engineer's  reaction  to  skill  vili  be  determined  in  part  by  the  number  of 
skill  levels  he  can  differentiate.  If,  for  example,  he  ccrsiders  skill 
to  have  only  two  levels  (i.e.,  skilled-unskilled),  the  range  of  his  design 
responses  to  a  skill  requirement  will  be  correspondingly  reduced. 

Data  describing  skill  are  ordinarily  presented  to  engineers  in  a 
variety  of  interactive  vays.  in  the  form  of  specific  requirements  {e.g., 
"design  so  that  equipment  can  be  operated  and  maintained  b;:  3-ieveI  per¬ 
sonnel"):  in  the  fora  cf  personnel  descriptions  (e.g..  Quantitative  and 
Qualitative  Fersonnel  Requirements  Information);  in  the  form  of  analyses 
such  as  task  descriptions  and  time-line  analyses;  in  the  fern  of  avail¬ 
ability  statements  (e.g.,  "highly  trained  personnel  from  the  X  system  will 
oe  made  available  to  operate  and  maintain  the  new  system");  in  terms  cf 
anticipated  training  time  and  costs.  It  is  therefore  necessary  to  deter¬ 
mine  vr.ich  of  these  types  of  skill  descriptions  are  most  readily  utilized 
by  engineers  and  in  particular  at  vhat  stage  of  system  development  they 
are  most  useful. 

Since  skill  is  not  the  only  design  parameter  considered  by  engineers 
in  making  design  decisions,  it  is  important  to  determine  the  weighting 
they  assign  to  skill  in  relation  to  other  parameters. 

Skill  is  ordinarily  viewed  by  behavioral  scientists  as  related  in 
some  way  to  the  amount  of  experience  that  personnel  have  had  in  the  same 
or  related  technical  specialty.  It  is  therefore  essential  to  determine 
whether  the  engineer  equates  various  levels  of  skill  with  different 
amounts  of  experience. 

If  &  relationship  exists  between  personnei  and  design  characteristics, 
it  seems  reasonable  to  assume  that  various  design  characteristics  will 
have  different  implications  for  the  amount  of  -.raining  needed  by  personnel 
to  achieve  proficiency  in  maintaining  equipment  with  these  design  char¬ 
acteristics. 

The  present  study  can  therefore  be  more  effectively  described  by  a 
series  of  questions  which  the  study  sought  to  answer.  These  are  simesarized 
as  follows: 


(1)  What  design  characteristics  and  design  concepts  are  considered 
by  engineers  to  be  related  to  skill  dimensions  and  skill  levels? 

(2)  How  many  Independent  design  concepts  can  be  differentiated  by 
engineers? 

(3)  What  priority  do  engineers  apply  to  design  parameters  (including 
skill  and  other  human  resources  data)  in  making  design  tradeoffs? 

(l»)  How  are  design  concepts  related  in  the  engineer's  opinion  to 
amount  of  training  required  by  these  design  concepts? 


(J?)  WAt  skill  dimensions  are  considered  by  engineers  to  differen¬ 
tiate  different  ievels  of  maintenance  technician  skill? 

(6)  Hcv  many  levels  of  skill  can  be  di fferentiated  by  engineers? 

(7)  What  is  the  relationship  in  the  engine*  r 's  mind  between  skill 
level  and  years  of  experience? 

The  ultimate  purpose  of  toe  present  study,  wnich  was  that  also  of  the 
previous  one,  is  to  derive  from  the  controlled  testing  of  engineers  certain 
personnel-design  relationships  vhic:.  would  enable  the  Air  Force  to  write 
more  effective  procurement  requirements.  It  is  assumed  that  if,  instead 
of  general,  non -enforce Hb le  manpower  provisions,  explicit  design-relevant 
statements  of  personnel  needs  can  be  incorporated  in  procurement  require¬ 
ments,  more  satisfactory  systems  will  be  developed. 


SECTION  II 


TEST  METHODOLOGY 


A.  GETZRAL  STRATEGY 

The  overall  research  strategy  involved  selecting  a  group  of  highly 
experienced  design  engineers  and  presenting  them  vith  a  series  of 
specially  developed  tests  which  examined  each  of  tne  questions  described 
in  tne  previous  section. 

As  part  of  another  study,  reported  in  Me is ter,  Sullivan,  Finley,  and 
Askren,  1969,  these  subjects  had  been  required,  immediately  prior  to 
entering  the  test  situation  described  in  this  report,  to  perform  the  con¬ 
ceptual  (paper)  design  of  a  missile  subsystem.  Since  the  inputs  provided 
in  this  design  simulation  included  the  equipment  and  HRD  inputs  normally 
provided  in  system  development,  and  since  they  were  required  to  make 
highly  explicit  design  decisions,  subjects  were  familiar  with  the  types 
of  personnel  inputs  and  design  problems  they  were  asked  to  examine  in 
the  present  test  series.  Since  the  earlier  HRD  inputs  were  specific  to 
the  design  problem  presented  in  the  previous  study,  whereas  the  test 
inputs  for  the  present  study  were  generalized  to  all  design,  subjects 
were  not  influenced  in  any  particular  response  direction  by  the  earlier 
inputs . 

It  appearel  feasible  to  apply  a  job  analysis  technique  to  the  inves¬ 
tigation.  A  set  of  skill  parameters  and  design  characteristics  -  as  large 
as  one  could  reasonably  conceptualize  -  was  developed  whicn  hypothetically 
differentiated  highly  skilled  maintenance  personnel  from  those  less  skilled 
(e.g.,  apprentices).  Sample  critical  incidents  Illustrating  these  para¬ 
meters  were  developed.  Design  engineers  were  asked  to  categorize  these 
incidents  and  parameters  in  two  ways:  as  (l)  they  conceptualized  skill 
differences  and  (2)  as  they  considered  that  skill  would  influence  their 
hardware  design. 

All  of  the  test  Items  were  administered  as  paper  and  pencil  tests  and 
given  tc  subjects  individually  in  two  four-hour  session- .  The  items  with¬ 
in  a  given  session  were  presented  In  a  randan  order  to  compensate  for 
serial  and  fatigue  effects.  Although  these  were  formal  tests,  they  were 
also  used  as  a  springboard  for  detailed  discussions  between  the  investi¬ 
gators  and  the  subjects.  The  general  procedure  was  to  administer  the 
individual  test  an?  then  follow  it  up  by  a  discussion  vith  the  subject 
concerning  the  nature  and  reason  for  his  responses,  before  going  on  to  the 
next  test.  This  permitted  more  meaningful  interpretation  of  the  subjects' 
responses. 
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B.  SUBJECTS 

The  eight  engineers  who  made  up  the  subject  population  for  this  study 
were  selected  from  the  test  engineering  department  of  the  Marquardt  Cor¬ 
poration,  Van  Nuys,  California.  Ergineerc  were  selected  from  this  company 
because  the  previous  study  (Keister  et  al,  1S&9)  had  required  the  selection 
of  personnel  skilled  in  the  design  of  test  equipment  used  to  check  out 
missiles  and  missile -related  equipment. 

An  analysis  of  the  education  and  experience  background  of  the  subjects 
is  presented  in  Table  I.  The  average  amount  of  experience  is  15*7  years, 
with  no  subject  having  less  than  8.  It  car.  be  said  therefore  chat  all 
subjects  were  highly  experienced.  Moreover,  they  are  essentially  equiva¬ 
lent  i\  terms  of  relevant  experience  to  those  of  an  earlier  study  (Keister 
et  al.,  1968),  In  which  the  mean  experience  level  was  17*3  years. 


C.  TEST  INSCRIPTION 

The  following  tests  were  developed  to  examine  each  of  the  study  ques¬ 
tions  asked  in  Section  I.  Tests  are  listed  in  Table  II  and  described  in 
terms  of  each  of  the  questions  they  sought  to  investigate.  The  tests 
themselves  and  the  instructions  given  subjects  are  presented  as  part  of 
the  Results  section  (III). 

1.  What  design  characteristics  and  design  concepts  are  considered  by 

engineers  to  be  related  to  skill  dimensions  and  skill  levels? 

In  order  to  structure  this  question  the  Investigators  differentiated 
between  design  characteristics  and  design  concepts  in  the  following  way. 
Design  characteristics  are  relatively  detailed  aspects  of  design,  such  as 
the  size  of  access  spaces,  or  the  placement  of  related  controls  and  dis¬ 
plays.  Design  concepts  are  combinations  of  design  characteristics  which 
describe  the  way  in  which  individual  equipment  characteristics  interrelate. 
For  example,  modular  packaging  (a  design  concept)  might  include  access 
space,  test  point  locftion  and  arrangement,  and  level  of  required  trouble¬ 
shooting  (each  of  which  is  a  design  characteristic).  Engineers  design 
equipment  in  terms  of  design  concepts,  such  as  level  of  automation,  and 
the  design  characteristics  relevant  to  those  concepts  fall  out  of  the  con¬ 
cepts;  however,  maintenance  men  respond  directly  to  the  individual  design 
characteristics . 

Skill  dimensions  are  different  ways  in  which  an  individual’s  skill 
(at  any  level)  can  be  manifested,  e.g.,  in  response  speed,  or  accuracy. 
Skill  levels  represent  varying  amounts  of  skill. 

In  Test  C  (Skill  vs.  Design  Characteristics  Test)  subjects  indicated 
whether  the  inclusion  of  various  types  of  characteristics  could  be  affected 
by  the  requirement  to  use  an  unskilled  man  as  operator  or  maintainer . 
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TABLE  I 


SUBJECT  EDUCATION  AND  EXPERIENCE 


Education 

Years  of  Experience 

BSEE 

12 

BSEE 

11 

BS 

17 

MSEE 

12 

BSEE 

8 

BSEE 

21 

BSEE 

22 

BS 

23 

IS.  7 

TABIE  II 


Test 


A 

B 

C 

D 


E-l 


E-2 


E-3 

F 

G 

H 

I 

J 


LIST  OF  TESTS  ALKINISTERED  TO  SUBJECTS 

Title  Relevant  Study  Question 

Skill  Dimens ions  I 

Skill  Importance  1 

Skill  vs.  Design  Characteristics  3 

Design  Tradeoff  Priorities  6 

Skill  vs.  Experience  Level  7 


Skill  vs.  Automation  3 
Skill  vs.  Design  Concepts  3 
Design  Independence  2 
Design  Suitability  3 
Skill  Ranking  1 
Design  vs.  Trt'ining  8 
Skill  Level  Differences  k 
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In  Test  E-2  (Skill  vs.  Automation  Test)  subjects  were  asked  to  indicate 
how  they  viewed  the  relationship  between  level  of  automation  and  skill  level. 

In  Test  E-3  (Skill  vs.  Design  Concepts  Test)  and  Test  G  (Design  Suit¬ 
ability  Test)  subjects  indicated  whether  a  limited  number  of  basic  design 
concepts  (as  differentiated  from  the  molecular  equipment  characteristics 
of  test  C)  were  relate-  more  to  skilled  than  to  unskilled  personnel. 

2.  How  many  independent  design  concepts  can  be  differentiated  by  engineers? 

In  order  to  reiate  design  characteristics  to  skill  levels,  it  is  neces¬ 
sary  first  to  establish  what  the  engineer  considers  to  be  independent  (i.e., 
homogeneous,  non-interactive)  design  concepts.  In  test  F  (Design  Independence 
Test)  subjects  were  given  fourteen  design  concepts  and  asked  to  determine 
which  of  these  are  related  or  independent. 

3*  What  priority  do  engineers  assign  to  design  parameters  (incl-ding  HRD 
inputs)  in  making  design  tradeoffs? 

This  question  had  been  asked  in  previous  studies  (Meister  and  Sullivan, 
1967,  and  Meister  et  al.,  1968).  It  was  considered  desirable  to  determine 
whether  the  present  subjects  utilized  the  same  design  tradeoff  orientation. 

In  Test  D  (Design  Tradeoff  Priorities  Test)  subjects  were  asked  to  rank 
twelve  design  parameters  in  terms  of  importance  to  design  decisions. 

What  skill  dimensions  are  considered  by  engineers  to  differentiate 
different  levels  of  maintenance  technician  skill? 


Test  A  (Skill  Dimensions  Test)  presented  a  series  of  statements  describ¬ 
ing  how  maintenance  technicians  perform.  Each  statement  was  an  incident 
descriptive  of  a  skill  dimension.  Subjects  were  asked  to  categorize  each 
statement  as  characteristic  of  a  skilled  or  unskilled  nan.  Those  statements 
on  uhich  significant  agreement  arong  subjects  could  be  reached  would  pre¬ 
sumably  describe  the  effective  parameters  of  maintenance  skill  as  defined 
by  the  designer.  The  following  tests,  dealing  with  the  same  question,  were 
also  presented: 

In  Test  B  (Sk-11  Importance  Test)  subjects  indicated  the  contribution 
of  each  of  nineteen  skill  dimensions  to  maintenance  skill  on  a  5  point 
scale. 

In  Test  H  (Skill  Ranking  Test)  subjects  were  enked  to  rank  the  nineteen 
skill  dimensions  of  Test  B  in  terms  of  their  importance  to  overall  skill. 

5-  How  many  levels  of  skill  can  be  differentiated  by  engineers? 

The  Air  Force  generally  utilizes  a  three -step  skill  level  continuum 
(3>  5>  7-level)  which  lacks  correlated  performance  descriritors .  It  was 
considered  possible  that  a  finer  skill  level  continuum,  phrased  in  terms 
of  characteristic  maintenance  performance,  could  be  utilized  more  readily 
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by  engineers.  The  purpose  of  Test  J  (Skill  Level  Differences  Test)  was 
to  determine  whether  subject  agreement  could  be  reached  on  the  seeps 
within  that  expanded  skill  continuum.  Subjects  were  given  a  10  point 
scale  of  skill  level  end  asked  to  assign  each  point  to  a  continuum  rang¬ 
ing  from  completely  unskilled  to  completely  skilled. 

6.  What  is  the  relationship  in  the  engineer's  mind  between  skill  level 
and  years  of  experience? 

Skill  level  and  experience  would  seem  to  be  somehow  related.  It  was 
considered  possible  that  some  quantitative  relationship  might  be  estab¬ 
lished  between  these  two  variables.  In  Test  E-l  (Skill  vs.  Experience 
Level  Test)  subjects  were  asked  to  assign  four  levels  of  skill  to  a  time 
scale  ranging  from  zero  years  experience  to  10  years  experience. 

7«  How  are  design  concepts  related  in  the  engineer's  opinion  to  the 
amount  of  training  required  by  these  design  concepts? 

A  relationship  between  design  concepts  and  amount  of  training  has 
commonly  been  assumed  in  most  manpower  planning,  but  the  nature  of  the 
relationship  has  not  been  investigated,  at  least  quantitative ly.  This 
question  was  an  attempt  to  scale  design  concepts  in  terms  of  amount  of 
required  training. 

In  Test  I  (Design  vs.  Training  Test)  the  design  concepts  studies  in 
Test  F  (Design  Independence  Test)  vere  given  to  subjects  with  instructions 
to  rank  each  concept  in  terms  of  the  amount  of  training  that  concept 
would  appear  to  require. 
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SECTION  III 


RESULTS  AND  CONCLUSIONS 


A.  SUWARY 

The  following  conclusions  were  derived  from  this  study. 

1.  Engineers  consider  (at  the  .001  level  of  agreement)  that  the 
following  design  characteristics  are  most  appropriate  for  skilled  per¬ 
sonnel:  quantitative  displays;  special  purpose  test  equipment;  trouble¬ 
shooting  down  to  the  circuit  level;  component  repair  required;  on-site 
maintenance  and  calibration. 


2.  Engineers  consider  (at  the  .001  level  of  agreement)  that  the 
following  design  characteristics  are  most  appropriate  for  unskilled  per¬ 
sonnel:  gc/no-go  displays,  step  by  step  procedures;  throw  away  mainten¬ 
ance  concept;  and  subsystem  as  opposed  to  component  checks. 

3.  The  following  design  attributes  are  considered  to  be  essentially 
independent  (i.e.,  unrelated  to  other  design  approaches);  unit  packaging; 
individual  packaging;  circuit  level  troubleshooting;  redundancy;  manual 
control;  and  scale  type  information.  Apparently  the  general  design  concept 
of  automatic  vs.  manual  equipment  can  be  broken  down  into  the  above  para¬ 
meters. 

h.  The  following  design  concepts  are  significantly  related  (in  the 
engineers’  mind)  to  skilled  personnel:  manual  control;  multi-purpose 
equipment;  circuit  level  troubleshooting. 

5.  The  amount  of  training  required  for  operation/naintenance  of  equip¬ 
ment  varies  as  a  function  of  design  attributes,  ranging  from  those  requir¬ 
ing  most  training:  circuit  level  troubleshooting,  multi-purpose  equipment 
and  scale  type  displays,  to  those  requiring  least  training:  self -check 
capability;  special  purpose  equipment;  and  go/no-go  information. 

6.  Engineers  have  a  ouch  more  precise  concept  of  what  constitutes 
skilled  maintenance  behavior  than  they  have  of  unskilled  behavior,  based 
on  the  consistency  of  their  responses  to  skilled  vs.  unskilled  character¬ 
istics. 

7.  All  eight  subjects  agreed  that  the  parameters  which  they  conceive 
of  as  differentiating  skilled  from  unskilled  maintenance  personnel  are: 


Characteristics  of 
Skilled  Personne.l 

Khowledge 


Characteristics  of 
Unskilled  Personnel 

Accuracy  (lack  o:') 
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Characteristics  of 
Skilled  Personnel 


Characteristics  of 
Unskilled  Personnel 


Problem-solving  Knowledge  (lack  of) 

(troubleshooting  ability) 

Flexibility  Problem  solving  (lack  of 

ability  to  troubleshoot) 

8.  The  following  parameters,  in  decreasing  order  of  importance,  are 
considered  to  contribute  most  to  skilled  performance:  integration  of 
information,  knowledge,  accuracy,  problem  solving,  training,  use  of  instru 
ments,  flexibility,  stress  performance,  visual  capacity  and  responsibility 

9<>  When  subjects  were  asked  to  rank  order  the  nineteen  skill  para¬ 
meters  in  terms  of  relative  importance,  seven  of  the  ten  parameters  noted 
in  (8)  above  were  included,  the  only  exceptions  being:  use  of  instruments 
stress  performance;  and  visual  capacity. 

10.  Engineers  consider  that  skill  is  less  important  for  determining 
design  characteristics  than  equipment  performance  capability,  equipment 
reliability,  cost,  maintainability  and  producibility. 

11.  There  is  no  consistency  in  engineers'  estimates  of  the  amount 
of  time  required  for  an  individual  to  achieve  various  skill  levels.  No 
significance  is  attached  by  the  design  engineer  to  time  on  the  job  as  an 
isolated  descriptor  of  ..kill. 

IE.  Engineers  claim  to  be  capable  of  differentiating  (at  the  .001 
level  of  significance)  ten  levels  of  personnel  skill. 


IE 


.i-  -  -  v.<  "  V 
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B.  DETAILED  RESULTS 


The  following  questions  were  answered  in  the  skill  investigation 
study.  Following  a  statement  of  the  question  a  description  (with  sub¬ 
ject  instructions)  of  each  test  is  given,  followed  by  a  statistical 
analysis  of  the  results  and  the  conclusions  derived  from  the  results. 

1.  What  desigr  characteristics  and  design  concepts  are  considered  by 
engineers  to  be  related  to  skilled  dimensions  and  skill  levels? 

Tests  C,  E-2,  £-3,  and  G  are  directed  toward  answering  this  question. 


TEST  C 

SKILL  VS.  DESIGN  CHARACTERISTICS  TEST 
Test  Description  and  Instructions 

In  this  test  we  wished  to  determine  how  the  engineer  relates  the 
skill  characteristics  of  the  personnel  who  are  going  to  operate  and 
maintain  his  equipment  to  the  characteristics  of  that  equipment. 

Subjects  were  presented  with  two  lists  of  characteristics:  one 
describing  the  personnel  who  would  be  responsible  for  operation  and 
maintenance  of  the  system  and  the  second  describing  the  characteristics 
of  that  equipment.  Originally  it  was  planned  to  have  the  subjects  con¬ 
sider  each  personnel  characteristic  (based  on  the  nineteen  parameters, 
which  would  have  made  nineteen  characteristics)  individually  against 
each  equipment  characteristic.  However,  the  number  of  comparisons 
required  quickly  exhausted  the  subjects  and  it  became  necessary  to 
collapse  the  individual  personnel  characteristics  into  three  which  de¬ 
scribed  (l)  a  relatively  low  skilled,  (2)  not  too  intelligent,  (3)  in¬ 
experienced  individual.  Subjects  then  judged  whether  or  not  a  partic¬ 
ular  design  characteristic  would  be  affected  by  the  requirement  to  use 
this  type  c.f  man  as  operator  or  maintainer. 

"One  of  the  things  we  want  to  find  out  is  how  the  engineer  relates 
the  skill  characteristics  of  the  people  who  are  going  to  operate  and 
maintain  equipment  to  the  characteristics  of  the  equipment  he  is  design¬ 
ing.  This  assumes,  of  course,  that  he  designs  his  equipment  to  fit  as 
much  as  possible  the  characteristics  of  its  users. 

"Imagine  then  that  you  are  designing  the  electronic  subsystem  of  a 
missile  or  a  maintenance  ground  equipment  and  that  you  have  a  description 
of  the  people  who  will  have  to  use  and  maintain  the  equipment.  These 
indi\ 'duals  can  be  summed  up  in  three  phrases:  (1)  relatively  lew  skilled; 
(2)  not  too  intelligent;  (3)  inexperienced.  These  characteristics  are 
constraints  on  your  design,  because  you  are  designing  for  relatively 
unskilled  personnel.  We  want  you  to  match  these  personnel  characteris¬ 
tics  to  the  characteristics  of  the  equipment  you  arc  designing,  for 
example,  the  weight  or  the  dimensions  of  the  equipment. 
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"We  will  function  something  like  this.  I  want  you  to  go  through  the 
list  of  design  characteristics  on  your  answer  sheet.  As  you  do  so,  ask 
yourself  whether  you  could  somehow  modify  the  design  of  that  equipment 
characteristic  to  make  it  easier  for  this  unskilled  technician  to  do  his 
job  better. 

"Just  as  an  example,  one  cf.  the  equipment  characteristics  listed  is 
"type  of  displays  -  quantitative  or  go/no-go".  You  might  feel  that  if 
the  technician  is  relatively  unskilled  he  might  not  be  able  to  assim¬ 
ilate  any  more  information  than  that  provided  in  go/no-go  displays;  so 
you  would  design  your  equipment  to  have  go/no-go  displays.  You  want  to 
ask  yourself  the  same  question  for  each  of  the  design  characteristics 
listed  on  your  answer  sheet.  That  sheet  ccntains  three  answers  you  can 
give  to  the  question;  no  effect;  slight  effect;  major  effect.  If  there 
is  no  possible  way  in  which  the  design  characteristic  could  be  modified 
to  fit  these  personnel  characteristics,  you  would  check  the  box  labeled 
"no  effect".  If  the  design  characteristic  could  be  modified,  but  only 
slightly,  you  would  check  the  box  labeled,  "slight  effect".  If  the 
design  characteristic  could  be  affected  significantly  bv  the  skill  char¬ 
acteristic,  you  would  check  the  box  labeled,  "major  effect".  Is  that 
clear? 

"Now  study  tne  list  of  equipment  characteristics.  If  you  have  any 
questions  at  any  time  about  what  you  are  supposed  to  do  or  what  each  of 
these  characteristics  means,  let  me  know.  You  may  wish  to  comment  on 
your  answer;  please  do  so.  Take  as  much  time  as  you  feel  you  need." 

Conclusions  and  Interpretations 

To  analyze  the  results  of  test  C  the  one-sample  test  (Siegel, 
1956)  was  used.  In  effect  this  s  '.tlstical  test  indicates  the  amount  of 
agreement  among  subjects  in  relating  skill  as  a  design  constraint  to 
particular  design  characteristics. 

An  examination  of  Table  III (a),  which  presents  the  frequencies  of 
subject  choices,  together  with  interpretative  comments,  reveals  that 
the  following  design  characteristics  are  significantly  related  to  lack 
of  skill:  arrangement  of  internal  components,  type  of  test  equipment 
required,  type  of  information  displays,  type  of  troubleshooting  symptoms 
and  step  by  step  procedures. 

If  the  two  categories  (slight  effect;  major  effect)  are  collapsed 
into  a  single  category  (some  or  major  effect),  (Table  111(b)),  the 
relationship  between  design  characterir  ics  and  skill  becomes  much 
stronger. 

Table  111(b)  indicates  the  following: 
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Design  Characteristics  Significantly 
(.05  level)  Belated  To  Low  Skill 
Level 


Test  points 

Internal  Components 

(a)  arrangement 

(b)  accessibility 

Checkout  procedures 

(a)  type 

(b)  length 

(c)  availability  of  feedback 

(d)  number  of  personnel  required 

Type  of  test  equipment  required 
Go/no-go  displays 

Troubleshooting  procedures 

(a)  symptoms 

(b)  feedback  indications 

(c)  level  of  troubleshooting  required 

Step  by  step  procedures 


Design  Characteristics  Significantly 
( .05  Level)  Unrelated  To  Low  Skill 
Level 

Weight 

Dimensions 

Type  of  equipment  connectors 
Drawings/ schematics 
Access  space 

Standardization  of  fasteners 


Those  design  characteristics  not  listed  above  are  aablguous  in  their 
relationship  to  skill  level;  in  other  words,  engineers  are  divided  in 
their  opinion  as  to  the  relationship. 

It  aust  he  emphasized  that  these  are  relationships  as  the  engineer 
sees  the*.  These  are  the  relationships  an  which  the  engineer  will  act 
is  his  design.  Since  they  are  .Judgmental  only,  they  should,  rot  he 
accepted  as  demonstrating  the  existence  of  an  actual,  empirically  derived 
relationship. 

The  implications  of  these  findings  is  that  should  the  developer  be 
required  to  design  for  low  level  personnel,  he  is  more  lixely  to  include 
in  his  equipment  the  design  characteristics  which  he  has  noted  as  being 
related  to  unskilled  performance.  In  addition,  there  would  seem  to  be 
a  definite  relationship  between  certain  classes  of  design  characteristics 
and  the  skill  leve.'s  of  the  personnel  expected  to  be  operating  that  equip¬ 
ment;  further  research  in  this  area  vould  seem  to  be  necessary  in  order 
to  examine  tne  relationship  of  major  categories  of  equipment  character¬ 
istics  and  skill  levels. 
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TABLE  ln(A} 

RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHARACTERISTICS) 

WITH  QRIGII J  L  CATEGORIES 
No  Slight  Major 

Hem  Effect  Effect  Effect  _  Responses 


1.  Weight  8 

2.  Dimensions  8 


Subjects  were  unanimous  in 
agreeing  that  equipment, 
weight,  and  dimensions  are 
not  related  to  skill. 


3.  Test  Points 

a.  Number  2 

b.  Location  2 

c.  Identification  2 

d.  Arrangement  2 

4.  Internal  Components 

a.  Type  4 

b.  Number  3 

c.  Arrangement  1 

d.  Accessibility  2 

5.  Checkout  I  rocedures 

a.  Type  2 

b.  Length  2 

c.  Frequency  of  Per-  5 

formance 


2 

2 

1 

1 


4  Subjects  were  divided  on 

this  question,  with  approxi- 

4  mately  half  of  them  indicat¬ 
ing  a  strong  relationship  be- 

5  tween  test  point  characterist¬ 
ics  and  skill. 


4 

2  3 

2  5 

2  4 


Here  again  we  have  an  al¬ 
most  equal  split  as  to  the 
relationship  of  personnel 
characteristics  to  the  layout 
and  selection  of  internal 
components. 


1  5 

2  4 

1  2 


The  nature  of  checkout  pro¬ 
cedures  is  seen  as  being  very 
much  a  function  of  personnel 
skill  and  experience. 


d.  Availability  of  Feed-  2  6 

back 


e.  Number  of  Personnel  *2  15 

Required 


1.6 


TABLE  III  (A) 


RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHARACTERISTICS) 

WITH  ORIGINAL  CATEGORIES 
No  Slight  Major 

_ Item _ Effect  Effect  Effect _ Responses 

6.  Equipment  Connections 
Required 


a.  Number  4 

3 

1 

Subjects  saw  very  little  rela¬ 
tionship  between  personnel 

b.  Type  7 

1 

characteristics  and  connector 
selection  and  utilization. 

Test  Equipment /Tools 

Required 

a.  Type 

Subjects  perceive  test  and 
checkout  philosophy  and  im¬ 

1.  Manual  /Automatic 

3 

5 

plementation  as  being  directly 

2.  Portable /Built- in  1 

4 

3 

related  to  the  characteristics 

3.  General  Purpose/ 

X 

c 

of  the  personnel  who  will  use 

Special 

the  equipment.  The  number  of 
test  equipments  required  does 

b.  Number  4 

2 

2 

not  seem  to  be  so  related. 

Test  Information  Displays 

a.  Number  3 

1 

4 

Subjects  seem  split  as  to  whe¬ 
ther  skill  level  is  related  to 

b.  Type 

number  of  displays.  There 
seems  to  be  no  doubt,  how¬ 

1 .  Quantitative  / 

C 

ever,  that  the  type  of  display 

Qualitative 

D 

provided  is  definitely  a  func¬ 

2.  Go/No-Go  1 

7 

tion  of  personnel  character¬ 
istics. 

Troubleshooting  Procedures 

a.  Symptoms 

The  nature  of  troubleshooting 
procedures,  the  level  to  which 

1.  Quantitative/ 
Qualitative 

n 

they  are  carried,  and  the  kind 

i 

of  feedback  required,  are 

17 


TABLE  HI(A) 


P 


Item 


RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHAR ACTERISTICS) 

WITH  ORIGINAL  CATEGORIES 

No  Slight  Major 
Effect  Effect  Effect 


Responses 


2.  Sing  1  e  /  M  u’tiple 

4 

4 

significantly  related  to  per¬ 
sonnel  characteristics,  with 

b. 

Number  of  Steps 

3 

1 

4 

number  of  personnel  required 
and  the  nature  of  symptom 

c. 

Feedback  Indications 

2 

1 

5 

presentation  being  somewhat 
less  closely  related. 

d. 

Number  cf  Personnel 

•a 

1 

A 

Required 

•J 

e. 

Level  of  Trouble¬ 

U 

shooting  Required 

L. 

u 

Type  of  Instructions 

a. 

Step-by-Step 

1 

7 

Subjects  report  that  step-by- 

Procedures 

* 

step  procedures  are  required 
for  unskilled  personnel.  How¬ 

b. 

Signal  Flow  Diagrams 

4 

2 

2 

ever,  drawings,  schematics, 
and  signal  flow  diagrams 

c. 

Drawings,  Schematics  7 

1 

apparently  cannot  be  tied  to  a 

particular  skill  level. 

Access 

a. 

Spaces 

The  number  and  size  of  ccess 
spaces  and  their  covers  a  "* 

l.  Number 

b 

2 

definitely  not  related  to  a  given 

2.  Size 

b 

2 

skill  level.  What  is  important 
is  simply  that  there  be-access. 

'♦)# 

Covers 

6 

2 

Coding  /  Labeling 

5 

1 

2 

Unrelated  to  skill  level. 

Standardization 

a. 

Components 

4 

1 

3 

Standardization  of  componm  ts 
and  fasteners  appears  not  to  be 

b. 

Fasteners 

8 

related  to  skill  level. 
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TABLE  HI  (A) 


RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHARACTERISTICS) 

WITH  ORIGINAL  CATEGORIES 


Item 


No  Slight  Major 

Effect  Effect  Effect _ Responses 


14.  Safety  4  3  1  ’’Safety  is  a  primary  consid¬ 

eration  at  all  times",  regard¬ 
less  of  personnel  characterist¬ 
ics.  Some  of  the  subjects, 
however,  would  hedge  this  a 
bit  by  paying  a  little  more 
attention  to  safety  for  uns.vTlto 
personnel. 
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TABLE  III  (B) 


RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHARACTERISTICS) 

WITH  COLLAPSED  CATEGORIES 


No 

Some  or 

Item 

Effect 

Major  Effect 

x2 

P 

1.  Weight 

16.  0 

.  001 

2.  Dimensions 

8 

16.  0 

.  001 

3.  Test  Points 

a.  Number 

2 

6 

7.0 

.  05 

b.  Location 

2 

6 

7.0 

.  05 

c.  Identification 

2 

6 

7.  0 

.  05 

d.  Arrangement 

2 

6 

7.0 

.  05 

4.  Internal  Components 

a.  Type 

4 

4 

4.0 

NS 

b.  Number 

3 

5 

4.  75 

NS 

c.  Arrangement 

1 

7 

10.  75 

.01 

d.  Accessibility 

2 

6 

7.0 

.05 

5.  Checkout  Procedures 

a.  Type 

2 

6 

7.0 

.  05 

bc  Length 

2 

6 

7.0 

.  05 

c.  Frequency  of  Per¬ 
formance 

5 

3 

4.  75 

NS 

d.  Availability  of 

2 

6 

7.0 

.  05 

Feedback 

e.  Number  of  Personnel  _ 

.  05 

Required 

i 

6 

7.  0 

i>.  Equipment  Connections 

Required 

a.  Numbe  r 

4 

4 

4.  0 

NS 

b.  Type 

*“» 

t 

1 

10.  75 

.  01 

7.  Test  Equipment/Tools 

Required 

a.  Type 

1.  Manual/ Automatic 

8 

16.0 

.001 

2.  Portable/ Built  - 

In  1 

7 

10,  75 

.  01 

3.  General  Purpos 

e/ 

8 

16.  0 

Special 

.  001 
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TABLE  HKB) 


RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHARACTERISTICS) 

WITH  COLLAPSED  CATEGORIES 


Item 


No  Some  or 

Effect  Major  Effect  _ _ P 


b.  Number  4 

8.  Tf  st  Information  Displays 

a.  Number  3 

b.  Type 

1 .  Quantitative  / 

Qualitative 

2.  Go/No-Go  1 


9.  Troubleshooting  Procedures 

a.  Symptoms 

1 .  Quantitative  /  ^ 

Qualitative 

2.  Single /Multiple  4 

b.  Number  of  Steps  3 

c.  Feedback  Indications  2 

d.  Number  of  Personnel  ^ 

Required 

e.  Level  of  Trouble-  ^ 

shooting  Required 

10.  Type  of  Instructions 

a.  Step-by-Step  Pro- 

cedures 

b.  Signal  Flow  Diagrams  4 

c.  Drawings,  Schematics  7 


11.  Access 

a.  Spaces 

1.  Number  6 

2.  Size  6 

b.  Covers  6 

12.  Coding /Labeling  5 


4 

4.  0 

NS 

5 

4.  75 

NS 

5 

4.  75 

NS 

7 

10.  75 

.  01 

7 

10.  75 

.  01 

4 

4.  0 

NS 

5 

4.75 

NS 

6 

7.0 

.  05 

5 

4.  75 

6 

7.0 

.  05 

7 

10.75 

.  01 

4 

4.0 

NS 

10.  75 

.  01 

2 

7.  0 

.  05 

2 

7.0 

.  05 

2 

7.  0 

.  05 

3 

4.  75 

NS 

2 


TABLE  113(B) 


RESULTS  OF  TEST  C 
(SKILL  VS.  DESIGN  CHARACTERISTICS) 

WITH  COLLAPSED  CATEGORIES 


Item 

No 

Effect 

Some  or 

Major  Effect 

x2 

P 

1 3 .  Standa  r  dizat  ion 

a.  Components 

4 

4 

4.  0 

NS 

b.  Fasteners 

8 

16.  0 

301 

14.  Safety 

4 

4 

4.  0 

IS 
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TEST  Fi-2 


SKILL  VS.  AUTOMATION  TEST 
Test  Description  and  Instructions 

In  this  test  subjects  were  presented  with  five  true/falcc  questions  ? 

regarding  the  relationship  between  level  of  automation  and  personnel  \ 

skill  level.  The  questions  are  presented  in  Table  IV.  . 

Conclusions  and  Interpretations  | 

The  Binomial  test  (Siegel,  1956)  was  applied  to  determine  whether 
the  distribution  of  true/faise  responses  varied  significantly  from  what 
one  might  expect  by  chance.  Four  of  the  five  items  are  highly  signifi¬ 
cant  . 

It  would,  seem  from  Table  IV  that  design  engineers  nave  a  pronoun¬ 
ced  bias  toward  designing  the  operator  out  of  their  equipment,  regard¬ 
less  of  personnel  requirements  levied  in  a  particular  situation.  Even 
item  C  in  the  table,  which  practically  "leads"  the  respondent  into  a 
response  favoring  a  manual  design,  fails  to  meet  significance  standards, 
although  five  of  the  subjects  did  agree  with  the  statement. 

Test  responses  are,  moreover,  corroborated  by  subjects  1  behavior 
on  the  major  design  problem  performed  in  the  earlier  sessions.  The  same 
behavior  has  been  observed  also  in  engineers  used  as  subjects  in  previous 
studies  (Meister  et  al,  1968). 


TEST  E-3 

SKILL  VS.  DESIGN  CONCEPTS  TEST 
Test  Description  and  Instructions 

"Indicate  whether  the  following  equipment  characteristics  would  be 
appropriate  for  or  could  be  used  more  readily  by  skilled  or  unskilled 
personnel.  Put  S  opposite  the  item  if  it  is  more  appropriate  ;or  skilled 
personnel;  U  opposite  the  item  for  unskilled  or  relatively  .nrki.led 
personnel.  Place  a  question  mark  opposite  the  item  if  i.,  i.o  applicable 
to  both  skill  levels  or  neither." 

Conclusions  and  Interpretations 

Twenty -eight  equipment  characteristics  were  preeente:,.,  .  :  v:  „ch 
twenty-two  reached  a  significance  level  of  agreement  of  .  jy  or  better. 
The  equipment  characteristics  applicable  to  unskilled  perc ..  • . re  , 
listed  in  Table  V(a);  those  applicable  to  skilled  personnel  ore  .-.sown 
in  Table  V'b).  The  six  characteristics  which  were  incigr.i; iacr.My 
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Beot  o 

Copy 


TABLE  IV 

RESULTS  OF  TEST  E-2 


(SKILL  VS.  AUTOMATION  TEST) 


T  F  P 

A.  All  other  things  being  equal,  when  system  require¬ 
ments  are  quite  stringent  (e.g. ,  high  reliability,  short  8  .004 

turn  around  time),  I  prefer  to  make  my  equipment 

more  automatic  than  manual. 

B.  All  other  things  being  equal,  if  an  equipment  re¬ 
quires  major  analyses  or  decisions  to  be  made,  1  7  1,  035 

prefer  to  automate  these  decisions,  unless  1  can  be 

sure  of  having  highly  skilled  personnel  to  operate 
and  maintain  the  equipment. 

C.  Certain  types  of  systems  (such  as  those  which  re¬ 
quire  sensing  or  interpreting  data)  seem  to  me  to 
require  more  manual  design  than  others. 

D.  Regardless  of  the  skill  of  the  personnel  who  will 

operate  and  maintain  the  equipment,  I  am  designing,  8  .  004 

I  prefer  to  minimize  the  possibility  of  error  by  de¬ 
signing  the  equipment  to  be  as  automatic  as  possible 

(all  other  things,  like  cost,  being  equal). 

E.  When  I  am  told  that  the  personnel  who  will  operate 

and  maintain  the  equipment  will  be  low  level  tech-  8  .  004 

nicians,  I  tend  to  make  my  equipment  more  manual 
(all  other  things,  like  cost,  being  equal). 


5  3  .  145 

NS 


TABLE  V  (A) 


RESULTS  or  TEST  E-3 
(SKILL  VS.  DESIGN  CONCEPTS) 

FOR  UNSKILLED  PERSONNEL. 


Distribution  of 
Responses 


Design  Concepts 

P 

Skilled  Unskilled 

Built-in  Test  Equipment 

10.  75 

.01 

1  7 

All  Similar  Components  in  One 
Module 

10.  75 

.01 

7 

1 

Go/NoGo  Displays 

16.  0 

.  001 

8 

Relatively  Short  Checkout  Pro¬ 
cedures 

10.  75 

.01 

1  7 

Internal  Components  Arranged 
by  Signal  Flow 

7.0 

.05 

6 

2 

All  Similar  Components  Grouped 
in  One  Module 

7.0 

.05 

6 

2 

Step  by  Step  Procedure 

16.0 

.001 

8 

Fixed  Maintenance  Equipment 

7.  0 

.05 

6 

2 

Throw  Away  Maintenance  Concept  lb.  0  .  QOi  8 

Subsystem  Checks  which  indicate 
that  the  Subsystem  is  either  Go/  16.  0 

No-Go 


001 


8 


TABL 

RESULTS  i 
(SKILL  VS.  UK 

FOR  SKILLE1 


r.e si/’U  Concepts 


C.  ^ntitative  Displays 

Jv  ’•iaole  Test  Equipment 

As  Much  Feedback  Informatio;. 

D. splayed  as  Possible 

i:u';  1  Purpose  Test  Equipment 

As  i  lany  Test  Points  as  Possible 

See pe  Pattern  Interpretation  Re¬ 
quired 

Trot  ;>leshooting  down  to  the  Circuit 

Level 

Component  Repair  Required 

On- Site  Maintenance 

Visual  Inspection  of  Components 

Required 

Li. i  Xumbe r  of  Maintenance 
Personnel 


Ca  •  :  ation  Required 


V  (B) 


OF  TES1  E-3 
SIGN  CONCEPTS) 

J>  PERSONNEL 


Distribution  of 
Respo 


P 

Skilled 

Unskilled 

7 

lb.  0 

.  001 

8 

7.  0 

.05 

6 

2 

10.  75 

.01 

** 

7 

1 

16.  0 

.  001 

8 

7.  0 

.  05 

6 

2 

10.  75 

.01 

1 

l 

16.0 

.001 

8 

16.  0 

.  001 

8 

16.  0 

.001 

8 

6.  25 

.05 

6 

l 

1 

10.  75 

.01 

7 

1 

If  0 

.  001 

8 

related  to  either  skilled  or  unskilled  personnel  are:  test  points  as 
close  as  possible  to  the  components  they  checkout;  internal  components 
stacked;  test  points  located  in  a  single  nodule;  long  checkout  procedures 
broken  up  by  stmmmry  u  Lotus  decks;  mobile  maintenance  equipment;  signal 
flow  diagrams  and  schematics. 

Test  E-3  is  related  tc  Test  C  (Skill  Design  Characteristics) 
and  supplies  somewhat  similar  answers.  Just  as  in  Test  C,  engineers 
are  likely  (all  other  things  being  equal)  to  provide  skilled  personnel 
with  quantitative  displays,  as  much  feedback  information  and  test  points 
as  possible,  to  require  troubleshooting  (component  repair  and  os -site 
maintenance  also  imply  troubleshooting)  and  calibration,  etc.  Design¬ 
ing  for  unskilled  personnel  implies  built-in  test  equipment,  go/no-go 
displays,  short  checkout  procedures,  step  by  step  procedures,  throw¬ 
away  maintenance  (thus  avoiding  the  need  to  troubleshoot),  etc. 

Again,  the  chances  are  that  requiring  that  equipment  be  designed 
to  either  high  or  low  skilled  personnel  will  predispose  the  engineer  to 
design  witn  tne  appropriate  related  characteristics.  This  is  not 
necessarily  bad,  since  there  is  a  logic  behind  the  relationship  of  skill 
and  design  characteristics.  However,  it  is  necessary  to  knew  what  the 
relationship  is  to  be  able  to  control  more  efficiently  the  engineer's 
design  tendencies. 


TEST  G 

EBSIGH  SUITABILITY  TEST 
Test  Description  and  Instructions 

"In  Test  F  we  gave  you  a  set  of  design  concept  descriptions.  How 
we  would  like  to  find  out  which  of  these  design  concepts  are  more  suit¬ 
able  for  skilled  than  for  unskilled  personnel,  and  vice  versa.  If  the 
concept  has  no  relation  to  skill  level,  place  a  check  in  the  "no  dif¬ 
ference"  column.  Bead  the  definitions  in  Test  F  over  again  and  place  a 
check  in  the  appropriate  column  below." 

Conclusions  and  Interpretations 

Table  VI presents  the  distribution  of  judgments  across  the  three 
categories.  If  the  null  hypothesis  is  correct  (that  there  is  no  dif¬ 
ference  among  the  frequencies  of  choice  among  skilled,  unskilled  and 
no  difference),  one  would  expect  a  frequency  of  3/3  in  each  column. 

This  permits  the  use  of  X2  as  a  measure  of  significance  betw»en  expected 
and  actual  values.  Table  X  indicates  that  4  of  the  14  cases  are  con¬ 
sistently  identified  as  having  same  relationship  to  skill  (at  the  .05 
level  or  better).  These  are: 


(1)  Manual  control  (skilled) 

(2)  Multi-purpose  equipment  (skilled) 

(3)  Circuit  level  troubleshooting  (skilled) 

(4)  Scale  type  information  (skilled) 

It  was  indicated  in  the  interpretation  of  Test  A  (Skill.  Dimensions) 
that  ei^jineers  have  a  more  precise  concept  of  skilled  than  unskilled 
behavior.  This  is  markedly  demonstrated  again  ir.  Test  G,  where  ail 
significant  relationships  refer  to  skilled  behavior. 

It  may  be  of  same  significance  that  three  of  the  design  concepts 
related  to  skill  are  among  the  "independent"  concepts  of  Test  P  (Design 
Independence ) . 


Discussion  of  Conclusions  Relative  to  Question  1 


It  is  apparent  from  the  preceding  four  tests  that  engineers  (con¬ 
sciously  or  not)  relate  individual  design  concepts  and  characteristics 
to  skilled  and  unskilled  behaviors.  Given  a  particular  skill  require¬ 
ment  the  engineer  Is  more  likely  than  not  to  include  in  his  design,  char¬ 
acteristics  which  he  considers  appropriate  to  that  skill  requirement. 
Table  V(a)  and  V(b )  list  the  characteristics  he  considers  appropriate 
to  particular  skill  levels.  Given  no  skill  requirement,  he  is  more 
likely  than  not,  all  other  things  being  equal,  to  automate  his  design. 

These  results  suggest  even  more  strongly  the  need  to  include 
specific  skill  requirements  in  procurement  specifications.  The  absence 
of  any  requirement  leaves  him  free  choice  to  indulge  his  biases;  and, 
as  certain  of  the  subjects  of  this  study  indicated,  he  tends  very 
strongly  to  design  in  accordance  with  ttese  biases.  It  would  seem 
desirable  to  control  the  engineer's  design  propensities  as  much  as 
possible  (however  much  this  may  be)  by  imposing  as  Aany  pertinent 
requirements  on  him  as  is  feasible. 


TABLE  VI 


RESULTS  OF  TEST  G 
(DESIGN  SUITABILITY  TEST) 


Design  Concepts 

Distribution  of  Responses 

No 

Differ- 

Skilled  Unskilled  ence 

x2 

P 

Complete  Automation 

1 

5 

2 

3.25 

NS 

Semi  -  Automation 

4 

2 

2 

1.0 

NS 

Manual  Control 

7 

1 

10.  75 

.01 

Self- Check  Capability 

1 

5 

2 

3.25 

NS 

Multi-Purpose  Equipment 

7 

1 

10.  75 

.01 

Special  Purpose  Equipment 

2 

3 

3 

.25 

NS 

Modularized  Equipment 

2 

4 

2 

1.0 

NS 

Unit  Packaging 

5 

3 

4.75 

NS 

Individual  Packaging 

5 

2 

1 

3.25 

NS 

Circuit  Level  Troubleshooting 

8 

16.0 

.001 

Module  Troubleshooting 

3 

3 

2 

.25 

NS 

Go /No -Go  Information 

1 

5 

2 

3.25 

NS 

Scale  Type  Information 

7 

1 

6.25 

.05 

Redundancy 

4 

1 

3 

1.75 

NS 
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2.  Sov  aaay  independent  design  concepts  can  be  differenti/ited  by 
engineers? 

Test  F  (Design  Independence  Test)  was  directed  at  answering  this 
question. 


TEST  F 

H5SIGH  IKLKFEJI1KHCE  TEST 
Test  Description  and  Instructions 

"In  this  series  we  want  to  explore  different  types  of  design  con¬ 
cepts.  Please  study  the  descriptions  of  equipment  characteristics  belov. 
Imagine  that  you  would  have  to  desi'  n  to  these  requirements .  Then  turn 
to  the  accompanying  matrix  table.  You  will  see  the  design  concepts  de¬ 
scribed  by  these  characteristics  listed  on  the  left  and  top  side  of  the 
matrix  table.  Pi  a  check  mark  in  each  cell  of  the  table  where  the 
design  approach  for  the  various  concepts  is  the  same.  For  example,  if 
the  design  approach  for  modularized  equipment  is  the  same  as  that  for 
an  equipment  with  self  check  capability,  you  would  put  a  check  in  the 
cell  where  these  two  descriptions  intersect.  If  the  design  requirements 
for  different  equipment  characteristics  are  different,  leave  the  cell 
blank." 

Complete  Automation 

All  functions  are  performed  automatically  by  the  equipment. 

The  operator  merely  monitors  its  performance. 

Semi  -Automation 


The  equipment  performs  functions  up  to  a  certain  stage,  then 
waits  until  the  operator  decides  to  go  to  the  next  functional 
stage  of  operation. 

Manual  Control 

The  equipment  performs  no  functions  without  the  operator  first 
initiating  a  control  action. 

Self -Check  Capability 

The  system  has  the  capability  to  check  its  own  functioning  and 
to  localize  its  faults  down  to  a  given  level. 


Multi-purpose  Equipment 

The  equipment  is  highly  flexible  and  performs  several  functions. 


imrowwinf mu  n  'i . [ ||f . — 
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Special  Purpose  Equipment 

The  equipment  is  specially  designed  for  a  particular  purpose 
and  performs  only  a  single  function. 

Modularized  Equipment 

The  Internal  components  of  the  equipment  are  packaged  In  a 
modular  fashion. 

Unit  Packaging 

The  equipment  has  individual  assemblies  to  perform  several 
different  functions;  however,  they  are  all  packaged  in  the 
same  unit. 

Individual  Packaging 

The  equipment  is  broken  up  into  a  nimber  of  individual  units 
which  can  be  hooked  together  In  different  arrangements  to 
checkout  different  assemblies. 

Circuit  Level  Troubleshooting 

Troubleshooting  of  this  equipment-  is  required  down  to  the 
circuit  level. 

Module  Troubleshooting 

Troubleshooting  of  this  equlpnent  is  required  only  down  to 
the  module  level. 

Go/lfo-Co  Information 

This  equipment  provides  information  to  the  technician  in  go/ 
no-go  fora. 

Scale  Type  Displays 

This  equipment  provides  information  to  the  technician  in  the 
form  of  quantitative,  scale  type  displays. 

Redundancy 

This  equipment  has  at  least  one  redundant  unit  or  path  for 
each  major  function  performed. 
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Statistical  Analysis 

The  subjects  made  one  of  two  possible  responses  for  each  matrix 
cell,  with  the  unbiased  probability  of  indicating  that  any  two  design 
concepts  are  equivalent  being  .5.  Therefore,  the  Binomial  Test  (Siegel, 
19^6)  is  applicable  and  the  specific  probabilities  for  selection  of 
two  design  concepts  as  being  equivalent  are  as  follows: 


Cell  Frequency 


Equivalent 

Hot- 

Equivalent 

Specific  Probability 

Frequency 

of 

Occurrence 

Yes 

Ho 

8  + 

0 

P  =  .004 

1^ 

7  + 

1 

P  =  .03^ 

16 

6  + 

2 

P  =  .l4j;  Hot  sig. 

24 

+ 

3 

P  =  -36  3  Hot  sig. 

2^ 

4  + 

4 

P  =  .637  Hot  sig. 

11 

TOTAL  MATRIX  CELIfi 

9- 

To  develop  a  measure  of  the  relative  dependence -independence  among 
design  concepts,  the  numerical  technique  of  squaring  the  element  values 
was  used  to  exaggerate  the  presence  of  cell  frequencies  greater  than 
one  (a  reasonable  cutoff  point  since  cell  frequencies  of  zero  and  one 
were  determined  (above)  to  be  nonchance  occurrences)  with  the  larger 
cell  frequencies  being  increasingly  emphasized.  The  sums  of  the 
squared  cell  frequencies  ("designed  by  equivalent  design  concept" 
responses)  yielded  an  ordinal  scale  describe  the  independence  of  the 
fourteen  design  concepts . 

Conclusions  and  Interpretations 

The  design  concepts  and  the  frequencies  of  selection  are  shown  in 
Table  VII.  Examination  of  Table  VII  reveals  that  roughly  one -half  of 
the  design  concepts  listed  can  be  considered  as  essentially  independent, 
independence  being  defined  as  those  items  which  few,  if  any,  of  the 
subjects  selected  as  being  related  to  any  other  design  concept.  The 
concepts  found  to  be  most  independent  by  the  above  described  measure 
are  listed  below  in  their  rank  order: 

(1)  Unit  packaging; 

(  2  )  Individual  packag ing ; 

(3)  Circuit-level  troubleshooting; 

(4)  Redundancy; 

(5)  Manual  control; 

(6)  Scale -type  information. 
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If  one  looks  at  those  concepts  which  are  related  to  other  concepts  (with 
a  frequency  of  7  or  6),  the  following  relationships  appear: 

(1)  Automation  and  semi-automation  are  related  to: 

(a)  self  check  capability; 

(b)  modularised  equipment; 

(c)  module  troubleshooting; 

(d)  go/no-go  information  (automation  only). 

(2)  Manual  control  is  related  to: 

(a)  multi-purpose  equipment; 

(b)  scale -type  information 

(3)  Self -check  capability  is  related  to: 

(a)  modularized  equipment 

'.4)  Modularized  equipment  Is  related  to 

(a)  module  troubleshooting* 

It  is  therefore  likely  that  if  the  designer  automates  his  equipment, 
wholly  or  partially,  he  will  tend  to  provide  a  self -check  capability, 
modules  and  go/no-go  information.  If  he  decides  to  design  his  equipment 
for  manual  operation,  he  is  likely  to  develop  multi-purpose  equips*. at 
and  present  quantitative  information  in  scale  type  displays. 
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TABLE  VII 

RESULTS  OF  TEST  F 
DESIGN  INDEPENDENCE  TEST) 


« 


1.  Complete  automation 

2.  Semi-automation 

3.  Manual  control 

■».  Self- check  capability 

5.  Multi-purpose  equipment 

6.  Special-purpose  equipment 
?.  Modularized  equipment 

8.  Unit  packaging 

9.  Individual  packaging 

10.  Circuit-level  troubleshooting 

11.  Module  troubleshooting 

12.  Go/No-Go  information 

13.  Scale  type  information 

14.  Redundancy 


gg- 

a 

sz- 
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Individual  packaging 
Circuit-level  troubleshooting 
Module  troubleshooting 


3.  What  priority  do  engineers  assign  to  design  puraaeters  ( including,  skill 
and  othir  HR1)  Inputs)  in  making  design  tradeoffs? 

Test  D  was  directed  at  answering  this  question. 


TEST  D 

DESIGN  TRADEOFF  PRIORITIES  TEST 

Test  Description  and  Instructions 

In  this  test  we  wished  to  determine  tno.  importance  of  HRD  inputs 
relative  to  other  design  parameters  in  any  tradeoffs  the  engineer  might  make. 
A  similar  test  item  has  been  included  in  a  number  of  previous  studies 
performed  by  the  authors  and  this  test  item  would  help  to  corroborate  or 
disprove  those  previous  findings.  The  value  essigned  by  the  engineer  to 
various  items  of  personnel  data  would  help  to  suggest  the  relative  weight 
which  should  be  attached  to  these  data  items  in  SCW's  or  in  human  factors 
inputs  provided  during  system  development. 

"Every  design  involves  a  number  of  tradeoffs.  Some  of  these  may  be 
more  important  tvan  others.  To  find  out  which  design  parameters  you  consider 
more  or  less  important  in  tradeoffs,  we  ask  you  to  rank  the  parameters  listed 
below  in  decreasing  order  of  importance.  Assume  you  are  considering  two 
alternative  designs  for  maintenance  ground  equipment.  The  decision  to  be 
made  is  to  be  determined  by  the  design  parameters  listed  below.  Which  of 
these  parameters  would  you  consider  most  important  in  terms  of  affecting 
your  design  decision? 

1.  Cost  of  developing  the  first  prototype. 

2.  Manpower  life  cycle  cost  -  the  cost  of  training  and  supporting  the 
crew  which  will  operate  and  maintain  the  equipment  from  the  time  the 
equipment  becomes  operational  to  the  time  it  is  inventoried  out  of 
service. 

3.  Equipment  reliability  -  probability  that  the  equipment  will  perform 
its  fm '  uions  when  required. 

Ij-.  Equipment  performance  capability  -  the  capability  of  the  equipment  to 
perform  to  Fission  requirements. 

Producibility  -  ability  to  produce  the  equipment  within  a  reasonable 
cost  and  schedule. 

length  of  training  -  length  of  ^ime  required  to  train  personnel  to  run 
the  equipment  as  required. 
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7.  Cost  of  training  -  cost  of  training  personnel  to  function  an  required. 

8.  Quantity  of  personnel  available  -  number  of  personnel  in  the  military 
inventory  available  to  man  the  equipment  being  designed. 

9.  Skill  availability  -  type  of  skilled  personnel  a\uilable  to  operate  and 
maintain  the  equipment. 

10.  Maintainability  -  ability  to  keep  the  equipment  in  an  "10)"  condition 
and  restore  it  to  functioning  status  vhen  it  has  malfunctioned. 

11.  Development  schedule  -  length  of  time  needed  to  deliver  the  first 
prototype  to  the  customer. 

12.  Personnel  skill  required  by  the  equipment  design  -  these  are  skill:., 
defined  in  terms  of  the  19  parameters  presented  in  series  B  (if  you 
have  forgotten  these,  please  review  their  definitions). 


"For  example,  one  design.  migit  increase  cost  but  reduce  the  length  of 
time  required  for  training  personnel.  Or  one  mlgrt  improve  producibility 
but  also  increase  cost.  You  have  to  decide  vhich  of  these  parameters  is 
most  important.  Rank  the  most  important  parameter  1,  next  most  important 
2,  etc." 


Statistic  1  Analysis 

The  ranks  assigned  to  the  various  desi  gn  parameters  are  summarized 
in  'Sable  VIII .  Kendall’s  W  statistic  (Siegel,  1956)  was  applied  to  the 
rankings  to  determine  subject  consistency.  W  =  .77,  which  is  significant 
at  the  .001  level,  indicating  that  the  subjects  were  almost  unanimous  in 
their  judgments.  The  results  of  this  test  corroborate  in  every  way  the 
results  of  similar  tests  administered  in  earlier  studies. 


Conclusions  and  Interpretations 


as  one  might  expect,  engineers  place  a  much  hi^ier  value  on  physical 
and  cost  parameters,  which  are  by  their  very  nature  restrictive,  than  they 
do  on  personnel  parameters.  All  the  personnel  items  are  ranked  below  -she 
physical/cost  parameters,.  It  is  however  interesting  that  the  most 
inqportant  personnel  parameter  to  eng?  ears  is  skill  level  required  by 
equipment. 

The  fact  that  engineers  assign  a  low  priority  to  personnel  inputs  should 
come  as  no  surprise  to  anyone  familiar  with  the  design  process  or  with  the 
preceeding  studies  of  engineers  (e.g. ,  Keister  and  Farr,  1966  and  Meister  and 
Sullivan,  1967^.  For  this  reason  it  is  important  that  additional  emphasis 
be  placed  on  timely  personnel  inputs  in  order  to  secure  a  more  equal 
distribution  of  the  engineer’s  attention. 


TABLE  VIII 


RESULTS  OF  TEST  D 
(DESIGN  TRADEOFF  PRIORITIES  TEST) 


Item  Rank 

Equipment  Performance  Capability  1 

Equipment  Reliability  2 

Cost  3 

Malnt  ^inability  4 

Froducibitity  5 

Development  Schedule  ^  ^ 

Personnel  Skill  Required  by  Equipment  Design  (Tie) 

Skill  A\ailability  8 

Manpower  Life  Cycle  Cost  9 

Quantity  of  Personnel  Available  10 

Length  of  Training  ^  ^ 

Cost  of  Training  (Tie) 
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How  are  design  concepts  related  in  the  engineer’s  opinion  to  the  amount 
of  training  required,  by  these  design  concepts'; 

Test  I  was  directed  toward  answering  this  question. 


TEST  I 

DESIGN  VS.  TRAINING  TEST 
Test  Description  and  Instructions 

"In  Test  F  we  gave  you  a  set  of  design  concepts.  We  want  you  to 
analyze  the  same  set  of  design  concepts  in  terms  of  the  amount  of  training 
that  would  be  required  to  enable  a  technician  to  operate  and  maintain  an 
equipment  designed  according  to  these  concepts.  Each  design  concept  has  been 
typed  on  a  separate  card.  Take  the  card  deck  and  arrange  u*e  cards  in  order 
of  the  amount  of  training  you  feel  would  be  required  for  that  type  of 
equipment.  The  card  cn  top  should  indicate  the  greatest  amount  of  training 
required,  the  second  card,  the  next  greatest  amount  of  training,  etc.*' 

Statistics ^  Analysis 

The  ranks  assigned  to  these  design  concepts  were  consistent  at  a  highly 
significant  level  (W  -  .41,  p  =  .001).  Table  IX  lists  the  design  concepts 
in  their  ranked  order. 


Conclusions  and  Interpretations 

It  would  appear  that  the  various  design  concepts  are  significantly 
related  to  the  amount  of  training  required.  It  is  interesting  to  note,  in 
view  of  our  earlier  hypothesis  that  troubleshooting  is  a  primary  dimension 
of  maintenance  skill,  that  circuit  level  troubleshooting  stands  almost  in  a 
class  by  itself  (mean  rank  of  1.6)  in  terns  of  the  amount  of  training  this 
design  concept  requires. 


It  would  be  highly  desirable  if  one  could  specify  in  a  procurement 
requirement  that  a  giver,  amount  of  training  was  to  be  provided  to  new  system 
personnel;  and  to  anticipate  that  consequently  ?.  particular  design,  concent 
would  be  employed  by  tne  engineer.  The  fact  that  such  relationships  car.  be 
drawn,  albeit  in  the  very  crude  manner  indicated  by  this  test,  suggests 
that  some  such  procedure  will  eventually  be  feasible.  The  relationship 
between  design  concepts  and  training  should  be  explored  further  because  it 
offers  substantial  potential  improvement  in  performance  efficiency  ar.i 
reduction  m  cost. 

Since  design  concepts  are  related  to  both  skill  level  ( Test  Z-)  ar.d 
training  (Test  I),  it  is  interesting  to  determine  the  extent  of  that 
co-relati onchip. 


in  Test  G  subjects  were  asked  to  describe  the  relationship  between 
design  concepts  and  skill  level.  In  Test  I  the  saae  design  concepts  were 
presented  to  engineers  and  they  were  asked  to  indicate  the  aaount  of 
training  required  for  each  design  concept. 

The  frequencies  across  the  three  categories  in  Test.  G  were  each 
weighted  with  respect  to  skill  level  and  suaned.  A  ranking  baaed  •-& 
skill  level  was  then  assigned  to  each  design  concept  on  the  basis  of  tne 
sun.  These  ranks  were  then  correlated  with  the  ranks  assigned  In  Test 
I,  using  the  S penman  rank-order  correlation  technique  (Siegel,  1956)* 

The  results  Indicate  a  highly  significant  level  of  agreement  (r„  -  .91, 
p  -  .01).  The  extent  of  the  agreement  appears  to  be  a  function  par¬ 
ticularly  of  those  itens  classed  as  requiring  skilled  maintenance  per¬ 
sonnel.  In  those  itens  of  Test  G  where  the  subjects  indicated  that  It 
■ade  "no  difference"  as  to  the  skill  level  of  personnel,  Here  was  a 
such  lower  consistency  in  subject  responses. 


TABLE  IX 
RESULTS  OF  TEST  I 
(DESIGN  VS.  TRAINING  TEST) 


Items 

Circuit  Level  Trouble  shootiz»g 
Multi- Purprse  Equipment 
Scale  Type  Display  s 
Individual  Packaging 
Unit  Packaging 
Manual  Control 
Redundancy 

Semi-Automatic  ) 

Module  Troubleshooting  J 
Complete  Automation 
Modularized  Equipment 
Self- Check  Capability 
Special  Purpose  Equipment 
Go/No-Gu  Information 


Mean 

Relative 

Amount  of 

Rank 

Rank 

Training  Rtq'd. 

1  -  c 

1 

Most 

4.  1 

2 

5.4 

3 

5.  7 

4 

5.9 

5 

6.  0 

6 

n  z 
•  •  J 

7 

8.6 

S.  5 

8.8 

9.6 

10 

9.7 

11 

10.  3 

12 

10.4 

13 

11.  o 

14 

Least 
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Test  Description  and  Instructions 


"The  following  is  a  set  of  cards  on  each  of  which  there  is  a  state¬ 
ment  which  describes  how  maintenance  technicians  perform.  Read  each 
statement  carefully  and  put  £  in  front  of  the  statement  if  you  believe 
it  is  more  characteristic  of  a  shilled  maintenance  nan.  Put  U  in  front 
of  the  statement  if  you  believe  it  is  more  characteristic  of  an  unski  lied 
or  less  skilled  nan.  If  you  lack  sufficient  information  to  make  a  Judge¬ 
ment,  or  you  feel  the  description  is  unrelated  to  skill,  put  a  question 
marie  beside  the  item.  You  must  categorize  each  statement;  do  not  miss 
any." 


(Shuffle  cards  for  each  subject.  Give  cards  to  subject  indivi¬ 
dually.  ) 


There  are  nineteen  parameters  and  two  descriptors  for  each  para¬ 
meter  (hence  thirty -eight  items}.  A  descriptor  is  an  incident  which 
describes  the  parameter  in  actual  performance.  One  descriptor  illus¬ 
trates  the  behavior  of  the  skilled  technician;  the  other  illustrates 
the  behavior  of  the  unskilled  technician. 


Each  parameter  is  listed  in  Table  X  above  the  two  descriptors 
which  represent  that  parameter.  (Subjects  did  not,  of  course,  see  tne 
parameter  categories,  which  are  presented  here  only  for  the  reader's 
benefit.)  One  descriptor  describes  the  parameter  as  one  sees  it  in  the 
skilled  man;  the  other  descriptor  describes  the  parameter  in  the  unskil¬ 
led  man.  ^The  descriptors  are  listed  in  Table  X  together  with  the  chi- 
square  (Xc)  value  indicating  degree  of  agreement  among  subjects  on 
that  descriptor. 


Statistical  Analysis 


Table  X  indicates  that  for  approximately  half  of  the  descriptive 
statements  the  subjects  agreed  at  a  highly  significant  level  (.01  or 
better)  that  these  kinds  of  behavior  are  capable  of  dletinguishii^  be¬ 
tween  skilled  and  unskilled  personnel.  Tue  test  used  to  determine  the 
degree  cf  agreement  among  subjects  was  the  Xs  one-sample  test  (Siegel, 
19^6). 
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TAB X  (Cont'd 


Has  been  working  as  a  maintenance  technician  o: 
electronic  eyatema  for  the  paat  5  yeara. 

Haa  worked  aa  a  maintenance  man  for  6  months 


TABLE  X  (Cont'd 
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■1 
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TABLE  x  (Cont'd. ) 


♦Skill  Parameters 


Coneiu-.lons  and  Interpretations 


Table  X  indicates  that  the  parameters  which  enter  into  the  deter¬ 
mination  cf  skill  are  a  function  of  vhich  significance  level  or  amount 
of  agreement  one  wishes  to  accept.  Tk_  .001  level  means  that  all  eight 
subjects  agreed  0.1  the  classification  of  a  descriptor;  the  .01  level 
means  that  seven  agreed  and  one  disagreed.  Accepting  those  C3scriptors 
which  are  significant  at  the  .01  level  we  find  that:  knowledge,  trouble¬ 
shooting  ability,  flexibility,  responsibility,  speed,  work  aids,  super¬ 
vision,  motor  coordination,  self-confidence,  innovation,  intelligence, 
and  visual  capacity  are  characteristics  of  a  skilled  technician;  while 
lack  of  accuracy,  lack  of  understanding,  and  inability  * j  troubleshoot 
describe  the  unskilled  man. 

Detailed  examination  of  the  items  on  which  all  subjects  agreed 
reveals  a  tantalizing  common  element,.  Almost  all  of  these  descriptors 
contained  a  reference,  explicit  or  implicit,  to  troubleshooting.  This 
suggests  that  the  commor  Actor  underlying  the  various  ways  in  which 
skill  or  lack  of  skill  l  ifest  themselves  in  the  maintenance  environ¬ 
ment  is  ability  to  trot  ^eshoot.  As  of  now  this  suggestion  can  be  hypo¬ 
thetical  only,  because  specific  questions  to  test  this  point  were  not 
developed.  It  is  interesting,  however,  that  Table  Xll(b)  indicates  that 
problem  solving  is  ranked  number  2  in  terms  of  the  components  that  enter 
into  maintenance  skill,  with  number  1  being  intelligence. 

In  analyzing  the  results  of  tills  test,  it  becomes  apparent  that 
those  behaviors  described  by  engineers  as  being  characteristic  of  skil¬ 
led  personnel  are  agreed  upon  with  much  more  consistency  (higher  level 
of  significance)  than  the  remaining  items.  Examination  of  the  eleven 
items  having  the  lowest  level  of  agreement  among  subjects  reveals  that 
nine  of  these  items  are  descriptions  of  unskilled  performance .  It  is 
possible  that  engineers  have  the  ability  to  recognize  what  is  obviously 
skilled  behavior,  but  are  unable  to  agree  as  to  what  constitutes  unskil¬ 
led  behavior. 


TEST  B 

SKILL  IMPORTANCE  TEST 


Test  Description  and  Instructions 

This  test  represents  another  way  of  probing  skill  dimensions.  In 
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Test  A  (Skill  Dime  ns:  nns ),  only  the  descriptive  incidents  were  presented, 
without  any  peraaeter  title  which  might  have  served  to  organize  the  con¬ 
cept  for  the  subject.  In  Test  B  the  parameter  title  was  presented  and 
defined.  In  addition,  the  subject,  indicated  the  contribution  of  the 
individual  parameter  to  skilled  maintenance  nerformanoe  on  a  5  point 
scale. 

While  it  is  impossible  to  directly  correlate  the  results  of  Tests 
A  and  B,  one  would  expect  to  find  the  most  important  parameters  of  test 
A  making  the  greatest  contribution  to  maintenance  skill.  Because  of  the 
manner  in  which  the  test  question  was  asked,  cue  would  also  expect  that 
even  parameters  on  which  there  was  no  significant  agreement  in  Test  A 
would  show  up  as  having  some  effect  on  sxill. 

"The  items  below  are  descriptions  of  the  various  ways  in  which 
maintenance  men  may  differ  in  terms  of  their  skill  level.  We  want  to 
find  out  which  characteristics  are  more  important  in  terms  of  the  way 
you  define  skill.  For  ease  of  presentation  we  summarized  each  skill 
parameter  in  a  short  phrase,  like  accuracy,  knowledge  or  motor  coordin¬ 
ation.  In  the  list  below  the  parameters  are  defined.  After  you  have 
studied  these  definitions  carefully,  we  will  ask  you  to  rank  each  skill 
description  with  regard  to  its  importance  in  maintenance  skill. 

"The  scale  runs  from  1  to  5  with  1  being  of  little  importance  and 
5  representing  a  highly  critical  component  of  maintenance  skill.  We  ask 
that  you  make  a  check  at  the  level  of  importance  that  you  think  repre¬ 
sents  the  Importance  of  that  skill  factor  in  maintenance  performance." 

LIST  OF  PARAMETER  IXFIHTTICBS 


1.  Integration  of  Information 

The  highly  skilled  maintenance  man  is  able  to  integrate 
information  from  different  quantitative  displays  (e.g.,  meters) 
and  to  interpret  its  meaning.  The  less  skilled  maintenance  man 
is  less  able  to  do  this. 


Minor 

Importance 

In 

Maintenance 

Performance 


12  3^5 

(Scale  was  repeated  for  all 
subsequent  items.) 


Critical 

To 

Maintenance 

Performance 


c.  Speed 

The  highly  skilled  maintenance  man  performs  maintenance 
checkouts  rapidly;  thr  less  skilled  man  is  slower  and  may  re¬ 
peat  steps. 


3.  aesponsibility 


The  highly  skilled  aan  performs  his  maintenance  tasks  by 
himaelf,  without  asking  assistance  of  others  or  needing  others 
to  monitor  him. 

h.  UOrk  Aids 


The  highly  skilled  man  usually  performs  his  tasks  without 
the  aid  of  checklists,  tech  manuals  or  written  Instructions; 
the  less  skilled  man  always  uses  checklists  or  other  instruc¬ 
tions  in  performing  his  task. 

5.  Accuracy 

The  skilled  nan  m&kf.s  relatively  few  errors  that  require 
him  to  repeat  his  work;  the  unskilled  man  makes  many  more 
errors  and  tends  to  repeat  his  work. 

6.  Supervision 

The  skilled  nan  either  supervises  others  or  is  referred 
to  by  others  for  assistance  or  approval  of  work  performed. 

The  less  skilled  man  does  not  supervise  or  assist  others,  al¬ 
though  he  may  be  able  to  do  his  own  work  satisfactorily* 

7.  Knowledge 

The  skilled  men  has  considerable  general  knowledge  (elec¬ 
tronic  theory,  maintenance  techniques)  as  well  as  specific 
information  about  his  equipment  * 

8.  Experience 

The  skilled  aan  has  worked  as  a  maintenance  technician  on 
this  and  similar  systems  for  a  number  of  years;  the  unskilled 
aan  has  much  fewer  years  of  experience. 

9»  Training 

The  skilled  ran  has  been  given  extensive  schooling  (for 
example,  advanced  schools  and  factory  training)  la  his  main¬ 
tenance  specialty;  the  less  skilled  aan  has  received  auch  lest 
formal  training. 

10.  Motor  Coordination 

The  skilled  aan  has  very  fine  manual  dexterity  and  is 
capable  o.  precise  wiring,  calibration  and  removing  and  re¬ 
placing  components  quickly;  the  unskilled  man  is  much  slower 
and  less  precise  in  his  mr-eaents. 
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The  highly  skilled  maintenance  man  is  capable  of  solving 
complex,  nonroutine  maintenance  problems  such  as  are  founi  in 
troubleshooting;  he  can  interpret  the  meaning  of  trouble 
symptoms;  the  less  SKtlled  man  has  difficulty  solving  these 
problems. 

12.  Self  Confidence 

The  highly  skilled  maintenance  man  has  no  hesitation  about 
volunteering  to  perform  more  complex  maintenance  jobs;  he  "seems 
to  know  what  he  is  doing."  The  les6  skilled  man  appears  to 
hesitate  when  starting  a  job,  as  if  he  we »<&-  not  quite  sure  of 
the  next  step. 

13.  'tee  of  Instnments 

The  highly  skilled  maintenance  man  te.uj  used  a  wider  variety 
of  test  equipment  (like  oscilloscopes),  and  is  familiar  with 
their  operations.  The  unskilled  man  i-  much  less  familiar  with 
and  has  used  these  Instruments  :t;ch  1* :s. 

14.  Innovation 

The  skilled  man  may  try  novel  approaches  to  the  solution 
of  a  problem,  such  as  procedural  a hr  ,*t  cuts .  The  unskilled  man 
follows  standard  maintenance  procedures  rigidly. 

15.  Rank 

The  skilled  man  has  higher  Air  Force  rank  than  the  unskil¬ 
led. 

16 .  Intelligence 

Tne  skilled  man  learns  nev  maintenance  procedures  quickly 
and  easily;  be  can  readily  relate  what  he  has  learned  tc  nev 
equipment.  The  less  skilled  man  learns  more  slowly  and  has 
difficulty  applying  what  he  has  learned  in  the  practical 
situation. 

17.  Flexibility 

The  skilled  .nan  can  perform  a  wide  range  of  maintenance 
jobs  and  switches  quickly  and  easily  from  one  type  of  job  to 
another.  The  less  Gkilled  man  has  difficulty  in  doing  so. 
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TABLE  XI 

RESULTS  OF  TEST  B 
(SKILL  LviPORTANCE-  TEST) 


Item 

Scale  Value  *  Rank 

x" 

P 

Integration  of  Information 

4.  5 

1.  5 

10.8 

.  05 

Knowledge 

4.  5 

10.  8 

.  05 

Accuracy 

4.  3 

3.  5 

13.25 

.02 

Problem  Solving 

4.  3 

10.8 

.05 

Training 

4-l1 

| 

10.8 

.05 

Use  of  Instruments 

4.  \  \ 

t  6.5 

23.  >5 

.001 

Flexibility 

4.!  j 

10.8 

.  05 

Stress  Performance 

4.lJ 

5.75 

NS 

Visual  Capacity 

4.  0 

9.5 

4.5 

NS 

Responsibility 

4.0 

10.8 

.05 

Intelligence 

3.7 

i: 

3.25 

NS 

Self-  Confidence 

3.  5 

12 

9.5 

.05 

Innovation 

3.  i 

13 

13.25 

.02 

Superviciv>i 

3.  1 

14 

10.8 

.05 

Motor  Coordination 

.CO*] 

5.75 

NS 

Speed 

3.  0  ’ 

r  16 

5.75 

NS 

Experience 

3.0j 

i 

f 

5.75 

NS 

Rank 

2.6 

18 

2.  0 

NS 

Work  Aids 

2.  3 

19 

15.  75 

.01 

*  Represents  mean  scale  value 


18.  Stress  Performance 

■"  -  i  ■  - 

When  required  to  work  more  quickly  to  meet  a  shortened 
deadline,  the  skilled  man  does  so  without  loss  of  efficiency; 
the  less  skilled  man  starts  to  make  errors  and  has  to  repeat 
himself. 

IQ.  Visual  Capacity 

The  r.killed  man  appears  to  have  keener  visual  perception 
than  the  less  skilled  man;  he  is  quicker  in  locating  components 
and  seeing  what  is  wrong  with  them  than  the  less  skilled  man. 

Conclusions  and  Interpretations 

In  twelve  of  the  nineteen  items  ranked  by  the  subjects  there  was  sig¬ 
nificant  level  of  agreement  (.05  or  better)  as  the  value  of  that  Item 
as  a  contributing  factor  to  maintenance  skill. 

fable  XI  indicates  that  of  the  four  highest  ranked  pa rate tors  in 
Test  B,  three  (knowledge,  accuracy  and  problem  solving)  were  the  same 
ones  considered  in  Test  A  to  differentiate  most  accurately  between  skil¬ 
led  and  unskilled  performance.  The  fourth  parameter  found  to  be  impor¬ 
tant  in  Test  A  (flexibility)  ranks  immediately  below  problem  solving  in 
Test  B„  Bence  the  results  of  Test  B  tend  to  confirm  the  results  of 
Test  A. 


«  TEST  H 

SKILL  HARKING  TEST 
Test  Description  and  Instructions 

“The  items  below  are  descriptions  of  the  various  ways  in  which  main¬ 
tenance  men  may  differ  in  terms  of  otter  skill  level.  Some  of  these 
ways  may  be  more  important  than  otters  in  determining  overall  mainten¬ 
ance  skill.  After  you  study  the  list  of  definitions  you  will  be  given 
a  deck  of  cards  on  each  of  which  one  definition  has  been  typed.  Please 
sort  the  dec",  of  cards  so  that  the  most  important  factor  is  on  top,  the 
next  most  important  is  second,  and  sc  forth  until  all  nineteen  cards 
have  been  ordered.  Do  not  leave  out  any  of  the  nineteen  cards.  If  you 
feel  that  any  of  these  factors  is  unimportant  in  determining  maintenance 
skill,  make  a  note  of  that  factor  and  tell  me  afterwards." 

Conclusions  and  Interpretations 

The  nineteen  parameters  presented  in  this  test  were  those  of  Test 
B  (Skill  Importance ) .  If  the  results  of  Test  B  are  compared  (Table  XII 
(b))  with  the  results  of  ranking  in  Test  H  (which  is  the  same  question 
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asked  in  a  different  manner),  we  find  that  there  ia  an  extremely  high 
correlation  between  the  two  rankings  (1  =  .71,  p  =  .01).  Kssentially 

what  is  represented  here,  in  a  hierarchical  order  of  importance,  is  the 
design  engineer's  concept  of  vbat  constitutes  the  skill  of  the  personnel 
for  which  he  is  designing  equipment. 

The  several  re1  ersals  in  rank  value  between  the  parameters  ranked 
in  Test  B  and  those  ranked  in  Test  H  are  explainable,  perhaps,  in  terms 
of  the  fact  tint  Test  3  is  asking  what  the  contribution  (in  an  absolute 
sense)  of  a  particular  parameter  is  to  maintenance  performance,  whereas 
Test  H  inquires  into  the  value  of  the  parameter  relative  to  all  the 
other  parameters  influencing  maintenance  performance,  nevertheless, 
parameters  ranked  2,  3,  II  and  5  in  Test  H  are  the  same  parameters  ranked 
highest  in  Test  B. 

Table  Xll(a)  indicates  the  ranks  assigned  in  this  test.  Kendall's 
W  (Siegel,  1956)  statistic  was  applied  to  determine  the  consistency  of 
subjects’  rankings.  The  V  value  derived  (W  =  .1*3,  p  -  .CCl)  indicates 
an  extremely  significant  level  of  agreement  among  subjects  relative  to 
skill  parameters. 


Discussion  of  Conclusions  Relative  to  Question  5 

The  fact  that  agreement  can  be  reached  among  engineer*  on  the  para¬ 
meters  entering  into  maintenance  skill  is  highly  encouraging  an/.'  should 
be  followed  further. 

Additional  investigation  should  be  made  of  the  hypothesis  that 
troubleshooting  ability  is  a  generalized  factor  influencing  the  engineer's 
concept  of  maintenance  skill  level.  It  may  be  possible  to  differentiate 
various  aspects  o-'  troubleshooting  and  thus  secure  a  more  precise  def¬ 
inition  of  maintenance  skill. 

In  any  event  it  would  appear  feasible  to  prepare  personnel  skill 
requirements  for  new  system  development  in  terms  of  the  more  highly 
ranked  skill  parameters.  Phrased  in  this  way,  such  skill  requirements 
may  be  more  meaningful  to  engineers  than  the  presently  employed  Air 
Force  Specialty  Code,  3,  5,  7  level  designations. 


53 


WBWWHfr'  ft* ' *  ‘ 


TABLE  XII  (A) 
RESULTS  OF  TEST  H 
(SKILL  RANKING  TEST) 


t 


Item 
Intelligence 
Problem  Solving 
Accuracy 

j  Knowledge 

Integration  of  Information 

\ 

Flexibility 

1 

:  Self-Confidence 

i 

i 

•  Responsibility 

Supervision 

i 

;  Training 

Use  of  Instruments 

s 

Experience 
Stress  Performance 
Speed 

Motor  Coordination 
Innovation 
Visua1  Capacity 
Work  Aids 
Rank 


fr 

* 


5 

} 


Rank 

1 

2 

3 

4 

5 

6 
? 

8 

9 

10 

II 

12.  5 

14.  5 

16 

17 

18 
19 


TABLE  «X  (*) 

RESULTS  OF  TEST  B  (SKILL  IMPORTANCE) 
COMPARED  WITH  TEST  H  (SKILL  RANKING) 


1-em 

Integration  of  Information 
Knowledge 
Accuracy 
Problem  Solving 
Training 

Use  of  Instruments 
Flexibility 
Stress  Performance 
Visual  Capacity 
Responsibility 
Intelligence 
Self-Confidence 
Innovation 
Supervision 
Motor  Coordination 
Speed 
Experience 
Rank. 

Work  Aids 


RANK 


Test  B 

1.5 

3.5 


6.5 


9.5 

11 

12 

13 

14 

16 

18 

19 


Test  H 

5 
4 
3 
2 

10 

11 

6 

12.5 

17 
8 
1 
7 

16 

9 

14.5 
14.  5 

12.5 
19 

18 
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6.  How  many  levels  of  skill  can  be  differentiated  by  engineers? 


Test  J  was  directed  at  answering  this  question. 

TEST  J 

SICELL  LEVEL  DIFFERENCES  TEST 
Test  Description  and  Instructions 

"The  following  items  describe  different  leyels  of  maintenance  skill. 

On  the  scale  below,  which  runs  from  completely  unskilled  to  completely 
skilled,  write  the  number  of  the  skill  level  item  where  you  think  that  level 
of  skill  belongs.  Note  that  the  skill  levels  described  are  not  necessarily 
equally  spaced  along  the  scale  and  you  do  not  have  to,  unless  you  want  to, 
space  them  equally." 

Skill  Levels 

1.  Supervises  trainees  in  performance  of  simple  maintenance  tasks. 

2.  Performs  all  mainterunce  tasks,  including  complex  troubleshooting, 
without  supervision. 

3.  Basic  schooling;  no  experience  at  all. 

4.  Responsible  for  performance  of  entire  maintenance  crew. 

5.  Allowed  to  perform  simple  maintenance  routine  on  his  own. 

6.  Learner;  acts  only  as  assistant  to  more  expeiienced  technicians. 

7.  Allowed  to  perform  simple,  routine  checkouts  under  supervision. 

8.  Performs  complex  maintenance  tasks  under  supervision. 

Supervises  less  skilled  men  in  troubleshooting  tasks. 

10.  Able  to  troubleshoot  to  the  circuit  level. 


Completely  Completely 

Unskilled  Skilled 


* 


* 


Conclusions  and  Interpretation 

Table  X3H  contains  the  ranks  assigned  to  each  of  the  items.  Using 
Kendall’s  "W"  statistic  (Siegel,  1956),  agreement  between  our  subjects  on 
this  item  was  extremely  significant  (W  «=  ,8,  p«  .001). 


Discussion 


From  the  results  of  ThhleXEUit  would  appear  that  engineers  can  differen¬ 
tiate  a  cumber  of  levels  of  maintenance  skill.  Although  there  is  some 
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overlapping  in  subject  rankings,  if  one  were  to  plot  their  distribution  in 
graphic  form,  it  would  approximate  a  straight  line.  This  suggests  that  the 
points  on  the  skill  continuum  described  by  the  various  skill  levels  in 
Dable  XI  represent  valid  stages  in  skill  progression. 

Earlier  studies  indicated  that  engineers  had  great  difficulty  in 
understanding  differences  among  the  Air  Force’s  3>  7-levels.  One  can 
conclude  from  this  test  that  engineers  are  capable  of  differentiating  a 
larger  number  of  skill  levels  when  these  sure  phrased  in  performance  terms 
than  if  they  are  phrased  in  terms  of  artificial  categories.  It  may  therefore 
be  viseful  in  specifying  manpower  requirements  in  procurement  documents  to 
describe  skill  levels  in  termr  of  performance  as  well  as  the  Air  Force’s 
AfSC  designations. 


7 .  What  is  the  relationship  in  the  engineer’s  mind,  between  skill  level 

and  years  of  experience? 

Test  E-l  was  directed  toward  answering  this  question. 


TEST  E-l 

SKILL  VS.  EXPERIENCE  LEVEL  TEST 


Test  Description  and  Instructions 

Test  E-l  was  designed  to  investigate  the  degree  to  which  engineers 
equate  experience  (time)  on  the  job  with  skill t  It  was  hoped  that  it  might 
be  possible  to  derive  some  skill/ experiential  standards  based  upon  the 
engineers  view  of  the  time/experience  continuum. 

"Assuming  that  the  technician’s  skill  is  related  in  some  way  to  the 
amount  of  experience  he  has,  and  without  considering  the  type  of  expe1^ er.ee, 
please  indicate  on  the  following  scale  whether  a  technician  with  certain 
years  of  experience  is 

1.  Unskilled 

2.  Slightly  skilled 

3.  Moderately  skilled 

k.  Highly  skilled 


Gradu¬ 

6 

1 

lfl 

2 

4 

3 

h 

5 

10 

ation 

oos. 

yr. 

mos. 

yrs. 

yrs. 

yrs. 

yrs. 

yrs. 

yrs. 

yrs. 

yrs. 

from 

exp. 

exp. 

exp. 

exp. 

exp. 

exp. 

exp. 

exp. 

exp. 

exp. 

exp. 

basic 

electronics 

school 


(Please  bracket  ()  on  the  scale  the  four  levels  of  skill  which  a  technician 
would  have  based  on  experience  alone,  all  other  things  being  equal.)" 


Statistical  Analysis 

Only  seven  subjects  responded  to  the  question.  Subject  number  eight 
refused  to  consider  the  possibility  that  length  of  time  on  the  job  was  by 
itself  in  any  way  related  to  skill. 

Hie  responses  of  our  seven  subjects  produced  the  following  averages 
and  ranges  for  the  four  categories  of  skill: 


I 

4 


{ 


1 
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TABLE  XIV 


RESULTS  OF  TEST  E-l 
(Skill  vs.  Experience  Level  Test) 


Item 

Average 

Range 

1.  Unskilled 

6.1  mos. 

1  mo.  -  12  mos. 

2.  Slightly  Skilled 

17.0  mos . 

3  nos.  -  30  mos. 

3.  Moderately  Skilled 

36.3  mos. 

6  mos.  -  60  mos. 

4.  Highly  Skilled 

36.3  mos. 

Conclusions  and  Interpretations 


Since  the  range  of  responses  is  so  great,  it  is  apparent  merely  from 
inspection  that  consistency  is  insignificant  and  further  statistical 
treatment  is  not  warranted.  Apparently  the  engineer  does  not  consider 
t.iae  on  the  job  by  itself  as  a  significant  descriptor  of  skill. 
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C.  CORCLUSIOHS 


1*  Design  characteristics  significantly  related  to  low  skill  level 
appear  to  be:  test  ^ints;  internal  components}  checkout  pro¬ 
cedures;  type  of  test  equipment  required;  go/no-go  displays; 
troubleshooting  and  step  by  step  procedures « 

2.  The  fcllowing  design  concepts  are  considered  by  the  engineer  to 
be  relatively  independent  entities:  unit  packaging;  Individual 
packaging;  circuit  level  troubleshooting;  redundancy;  manual 
control  and  scale  type  information. 

3.  Design  concepts  and  characteristics  are  related  in  the  engineer's 
mind  to  different  skill  levels.  Given  a  particular  skill  require¬ 
ment,  the  engineer  is  more  likely  than  not  to  incl  .  in  his 
design  characteristics  that  he  considers  appropri.  '£>  that 
skill  requirement. 

4.  The  engineer  conceptualize r  maintenance  skill  in,  terms  of  know¬ 
ledge,  troubleshooting  ability  and  flexibility;  lack  of  skill  is 
described  by  lack  of  accuracy,  failure  to  understand  and  inability 
to  troubeshoot.  A  common  denominator  of  skill  parameters  appears 
to  be  troubleshooting  ability. 

5.  The  engineer  has  a  concept  of  skill  level  which  la  more  operation¬ 
ally  and  performance -oriented  than  described  by  the  Air  Force 
Specialty  Code  designations. 

6.  The  individual  design  concepts  are  significantly  related  in  the 
engineer's  mind  to  the  amount  cf  training  required.  A  specifi¬ 
cation  of  the  amount  of  training  to  be  given  to  system  personnel 
will  therefore  influence  the  type  of  design  concept  Incorporated 
into  new  equipment. 

7.  Skill  level  appears  not  to  be  .'elated  in  the  engineer's  mind  to 
years  of  experience. 
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SECTION  IV 


RECOMMENDATIONS 


Tn  urder  to  influence  system  design,  engineers  must  be  provided  with 
personnel  requirements  which  are  meaningful  to  them  in  terms  of  design 
implications.  Manpower  requirements  usually  incorporated  in  procurement 
documents  do  not  satisfy  this  criterion.  The  reason  for  this  is  the  serious 
lack  of  data  on  personnel-design  relationships.  The  present  study  has  made 
a  very  tentative  stab  at  uncovering  these  relationships  and  it  is  encouraging 
to  note  that,  under  controlled  conditions  (as  in  this  study)  engineers  appear 
to  recognize  the  validity  of  these  relationships.  Much  more,  however, 
remains  to  be  done.  Later  in  this  section  reconmeodations  for  further 
research  will  be  presented;  for  the  moment  the  immediate  implications  of 
the  study  should  be  considered. 

As  a  result  of  the  present  study  and  others  performed  (eg,  Keister 
et  al. ,  1968  and  1969)*  it  is  apparent  that  engineers  will  disregard  man¬ 
power  requirements  phrased  in  general  terms  such  as,  "design  to  minimize 
skill  level".  It  is  therefore  necessary  for  the  human  resources  specialist 
to  include  in  procurement  documents  hignly  explicit  requirements  which  have 
the  following  characteristics: 

(1)  Manpower  requirements  should  be  phrased  as  explicit  design 
requirements. 

(2)  Indicate  the  maximum  number  of  personnel  allowed  for  operation  and 
maintenance  of  the  system. 

(3)  Indicate  the  distribution  of  skill  level  of  these  personnel. 

(4)  Indicate  the  amount  of  training  which  will  be  given  these 
peisonnel  and  the  level  of  proficiency  they  can  be  expected 
to  attain. 

(5)  Skill  level  should  be  phrased  in  performance  terms  (ie,  in  terms 
of  what  the  personnel  can  be  expected  to  do  while  operating  and 
maintaining  the  system). 

(6)  The  design  implications  of  manpower  requirements  should  be 
explicitly  stated  as  desired  outputs  of  system  design  (eg, 
personnel  should  (cr  should  not)  he  expected  to  troubleshoot 
at  the  circuit  level). 

The  inclusion  of  suck  manpower  requirements  in  procurement  documents 
will  not  necessarily  cause  the  engineer  to  give  these  requirements  the 
highest  priority  in  his  design,  but  they  will  cause  him  to  think  more 
seriously  of  his  design  in  terms  of  personnel  implications. 
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She  study  also  suggests  ceitain  fruitful  areas  for  further  work.  The 
nature  of  equipment  design  makes  particular  demands  upon  personnel  in  terms 
of  the  skills  needed  to  operate  and  maintain  that  equipment.  Prom  that 
standpoint  it  is  necessary  to  analyze  tne  various  equipment  designs  avail¬ 
able  or  projected  to  become  operational  in  tho*  Air  Force  to  determine  the 
tasks  they  inpose  on  personnel  and  the  skills  they  will  require.  As  new 
equipment  enters  the  inventory,  new  skills  may  well  be  required;  studies  of 
present  equipment  may  suggest  the  new  skills  tfhich  may  be  demanded.  In  any 
event,  skill  as  a  dimension  of  personnel  performance  cannot  be  studied  apart 
from  the  operational  context  in  which  that  skill  must  be  utilized. 

Since  the  essence  of  the  problem  seems  to  be  to  relate  manning  and 
skill  level  to  concrete  design  concepts  and  characteristics,  it  is  suggested 
that  this  area  have  higi  priority  in  further  explorations.  The  present 
study  has  suggested  certain  relatic.iships ,  but  these  deal  only  with  e 
particular  type  of  maintenance  equipment.  It  is  recommended  that  an  an 
analysis  be  made  of  deferent  types  of  operator  and  maintenance  systems  and 
that  the  manning  and  skill  dimensions  pertinent  to  the  design  characteristics 
of  these  systems  be  uncovered. 

Such  a  study  could  be  performed  by 

(1)  Developing  a  taxonomy  of  design  concepts  and  characteristics  which 
differentiate  various  types  of  qperator/maintenance  systems,  eg, 
airborne,  ground,  coemand/ control ,  maintenance,  esc. 

(2)  Selecting  various  operational  systems  in  the  Air  Force  inventory 
which  are  representative  of  the  various  clauses  of  design  concepts 
and  characteristics  within  the  taxonomy  of  (l)  above. 

( 3)  Going  to  the  air  bases  at  which  these  systems  are  to  be  found  and 
performing  an  analysis  of  the  various  task  behaviors  required  by 
the  different,  design  concepts/characteristics. 

(4)  Determining  from  the  operators / aaintenanc e  technicians  who 
utilize  the  equipments  the  various  manning/ skill  parameters 
involved  in  these  behaviors.  Hie  design-skill  relationships 
uncovered  in  the  present  study  would  be  investigated  in  operational 
performance  and  correlated  with  the  results  of  the  present  study. 
This  might  require  the  development  of  and  administration  to 
operational  personnel  of  tests  to  uncover  the  manning/skill  para¬ 
meters  which  differentiate  effective  from  non-effective  performance 
on  particular  types  of  equipment.  Additional  data  would  he 
gathered  relative  to: 

(a)  the  type  of  training  provided  to  personnel  of  the  various 
systems. 

(b)  the  kinds  of  personnel  performance  problems  experienced  in 
operating  and  maintaining  these  systems. 
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(j)  The  end  produet  of  this  study  would  be  the  development  of  what 

can  be  termed  a  "human  resources  data  system  development  handbook", 
consisting  of  descriptions  of  the  various  classes  of  equipment 
and  their  design  characteristics,  together  with  the  manning/skill 
dimensions  required  to  operate  and  maintain  that  equipment, 
applicable  training  data,  characteristic  task  behaviors,  etc. 

Such  a  handbook  could  then  be  provided  to  Air  Force  managers  and  design 
engineers,  so  that,  given  a  requirement  to  develop  a  system  of  a  given  type, 
they  could  look  up  the  appropriate  equipment  design  type,  and  find  the 
manning  diversions  required  for  that  equipment.  Such  a  handbook  could  be 
updated  periodically  as  nev  types  of  equipment  (e.g.,  micro-electronic  cir¬ 
cuitry)  enter  the  inventory.  Appropriate  parts  of  the  handbook  could 
be  referred  to  in  procurement  specifications  for  new  equipment  as  a  guide 
to  the  manager  and  engineer. 

What  has  been  described  is  not  simply  smother  form  of  job  analysis, 
although  it  contains  elements  of  such  sun  analysis,  but  a  highly  design- 
oriented  catalogue  which  could  be  gradually  extended  to  cover  the  totality 
of  equipment  types  used  by  the  Mr  Force.  Since  the  Air  Force  expects 
engineers  to  consider  manning  requirements  in  their  design,  it  would  seem 
only  reasonable  to  supply  the  engineers  with  the  appropriate  information 
they  need  to  consider  these  requirements. 

The  utility  of  such  a  handbook  ^ould  be  tested  by  utilizing  the  seme 
research  strategy  employed  in  the  present  study,  that  is,  by  giving  a 
sample  of  engineers  the  catalogue  to  use  in  the  course  of  performing  a 
simulated  design.  Such  tests  would  serve  as  feedback  to  enable  the  Air 
Force  to  supply  the  kind  of  data  best  suited  to  engineering  needs  for  new 
system  development. 
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